After rewriting SQLite in Rust, Turso turns its sights on Postgres

Having built a SQLite-compatible database from scratch in Rust, cloud database provider Turso is attempting something more ambitious: a Postgres-compatible implementation on the same foundation. "I don't think there is anything fundamentally wrong with Postgres. If there was something wrong with Postgres, we wouldn't rewrite it …

[VoiceOfTruth]

Freshened up

>> But, like SQLite, Postgres could be freshened up for the cloud-native era.

Newspeak for newly written code, without decades of testing behind it. Yeah, I'll jump on that bandwagon.

[Anonymous Coward]

Re: Freshened up

"I don't think there is anything fundamentally wrong with Postgres" - agreed. It is rock solid, kept up to date and free. I haven't needed to look to any other relational storage for the last 15 years or so. Which is why I'll be ignoring this rewrite that has few tangible benefits but many potential risks.

[Cliffwilliams44]

Re: Freshened up

You missed the 'need' in the article, Postgres; 'one process per connection; that is not only expensive computationally in a cloud environment it is expensive is money!

If they can solve this, it's a real advantage.

This isn't a rewrite of Postgres to replace Postgres; it's a Postgres compatibility layer on top of the Turso engine. Their goal is to have more of these, MySQL, MSSQL, etc. Your prejudice, like so many in this forum blinds you to this.

[Rich 2]

Re: Freshened up

But the “one thread per connection” issue seems to be a “…by the way…” comment rather than a justification for rewriting it. And I’m sure fixing this issue could be done without a rewrite in a different language

And why rewrite sqlite?

I’m not really having a go at anyone - if someone wants to rewrite something then that’s up to them - I’ve done it myself. But rewriting something on top of the gloop of some common platform code that wasn’t there before and then “selling” it as the next shiny without any real benefit (and probably with its own bugs - would you trust YOUR data to it?) doesn’t make any sense to me. Seems to be a solution (in fact it’s not even that) to a non-existing problem

[that one in the corner]

Re: Freshened up

> Their goal is to have more of these, MySQL, MSSQL, etc. Your prejudice, like so many in this forum blinds you to this.

Blinded by prejudice? Prejudice against what? Against just grabbing something new for no good reason?

If the Turso engine manages to demonstrate some practical advantage over SQLite, then great, good for them, will look at the cost of giving it a try. Ditto Postgres* or any other db they manage to mimic, if that is relevant.

* and note, not all installations of Postgres give a monkey's about the process-per-connection, it just ain't relevant. Your use-case may be different: bully for you. Having different use-cases is not evidence of prejudice.

[Detective Emil]

Gotta break 'em all!

Yeah, let's put all our DBMs on top of a common engine so that a bug in the engine can floor the lot of them.

[Bebu sa Ware]

Re: Gotta break 'em all!

> Yeah, let's put all our DBMs on top of a common engine so that a bug in the engine can floor the lot of them.

Putting all your eggs in one basket but as I think Bruce Schneier noted in relation to security eggs, you keep a very close eye on that one basket.

If your common virtual database engine were formally defined and formally verified you might well be in a better position.

All RDBMS fundamentally do much the same things only slightly differently and (often gratuitously) incompatibly which ultimately converge on a layer of abstraction like Turso. Having a separate abstraction for RDBM authentication and authorisation might also make sense — A&A never seemed to be one of their better features.

Always struck me that databases were like operating systems (kernel's resource and concurrency management mostly) … although very shitty operating systems.

[FIA]

(much like Andrew Tanenbaum kept tight control on Minix, inspiring a young Linus Torvalds to create a much more ambitious Linux).

Ambitious?

One is a microkernel, with drivers in user space and process comms with message passing primitives, written by someone who (literally) wrote the book on operating system design.

The other is a UNIX kernel knockoff built to run the GNU userland.

Linux is amazing, and what it has done for computing should never be understated, but in the scope of OS development it's not in any way ambitious. It's a re-write of what was then a fairly old kernel design. Arguably it's held back development in some areas. (I mean we've still not got a decent driver layer.... I still shudder to think how many hours of programmers time that's wasted...)

When SATA drives were released, you could load a driver from a floppy to make them work on the already released Windows XP. You still can't do that on Linux to this day.

Please don't misunderstand this comment, it's not meant as a slight against Linux or Linus, I could not do what he has done. I just find OS internals very interesting and I'm always a bit triggered when I hear Linux presented as something other than a UNIX kernel knockoff.

Linux's influence is not to be understated, but from a purely technical point of view it was more right place/right time than anything else. (HURD wasn't ready, the BSDs were potentially problematic with regards to copyright).

[williamyf]

You are 107% right.

The most interesting OS nowadays, from a kernel and new ideas PoV are QNX, Minix 3 and the L4 family of µKernels.

NT could also be interesting, if microsoft put the current semi-monolythic kernel on a diet and returned the kernel back to its µkernel roots, but we know that aint'-gonna happen.

[Anonymous Coward]

VxWorks is sliding ever more towards UNIX, to make it more approachable to a wider audience. Or so say the colored pencil department.

[bazza]

In a way, it's been a "Unix" for a looooong time. It's had a POSIX layer for decades. I've had X-Server running on it for 30 years, with colour! It's native shell is definitely not a Unix shell, but there's not been much stopping one compiling up a sh if one wanted to.

One of the strengths of VxWorks I always felt was that it wasn't a Unix under the hood, in the sense that it didn't provide process separation. From an embedded real time system point of view, and especially when having to hack around inside one at arms length (e.g. earth to Mars), VxWorks' lack of process separation has enabled some truly epic saves. I don't know of any other operating system that provides the features of a full-up multitasking pre-emptive OS with MMU support, assured scheduling, etc. that has a wide open memory space across the whole system. It allows one to approach an embedded system a bit like it's a proper grown up microcontroller++ with proper memory management - a rarity.

I know they've since added a process separation to VxWorks, which no doubt suits some customer applications, but then it's kinda a run of the mill operating system again.

[bazza]

Those are old ideas that are possibly receive new attention, as folk discover they exist. To those that know these things of old, their benefits are familiar...

[Anonymous Coward]

I mean you can load drivers from an external block device. They either have to be built for that extract kernel already, or they are done with DKMS to be build on system. What this comes down to is the old Unstable ABI debate. Which Linux source a write up about : https://docs.kernel.org/process/stable-api-nonsense.html

[bazza]

Ironically, in some Linux distros the GNU userland is being replaced by the equivalent re-written in Rust...

The lack of a stable approach to device drivers feels insane to many (including me!). But it's the kernel community's choice. There are external pressures on that, not least mobile phones (Google has already taken some independent action on this, effectively inserting a stable driver layer for the benefit of device manufacturers). Rust is an external pressure too, which they're finding increasingly difficult to ignore. History tells us that projects that completely fail to flow with the times eventually wither; in this world - even a non-commercial OSS world - it's a case of develop or die. One feels that if the Linux kernel project wholeheartedly embraced Rust and finally did stabilise its driver interface, the project would be set fair for a very long time to come.

Certainly, if they did announce "we're going to convert the whole kernel to Rust", there'd be a lot of interest from many parties in that endeavour. Whole governments would be keen for a start, and might shake loose some resources and assistance.

Linus did get lucky in some regards. The FreeBSD project is another close contemporary that hit the streets just a short while after Linus released his kernel. Had that come first, it's possible that it may have won.

[steelpillow]

Reality check

Some of you guys need to read Eric Raymond's The Cathedral and the Bazaar - the whole book, not just the eponymous essay.

Linux did two magic things - he adopted the GNU copyleft, and he groomed his contributor community.

Oh, and according to his autobiography Just for Fun, he stared out from Minix as a base. Some whiners still complain it was Minix he plagiarised and not Unix. Truth is, he cobbled up a barely functional proof-of-concept and persuaded the rest of the world to make it work properly. And he's still doing that.

[Taliesinawen]

Linux and SATA drivers

> (I mean we've still not got a decent driver layer.... I still shudder to think how many hours of programmers time that's wasted..

Linux has supported loadable kernel modules for decades.

[EvaQ]

Re: Linux and SATA drivers

> Linux has supported loadable kernel modules for decades.

Yes, as part of the kernel, which need to be compiled for that kernel. Realtek can't give a binary driver that will work for "Linux" (like they can for Windows). And if they give source code, it's not working for the user right away: the user has to included & compile it, or the distribution maker, or the kernel makers.

[Anonymous Coward]

They should have called it pgtips

[Anonymous Coward]

They should have called it pgtips

Watered down, cloudy rusty-coloured Postgres? Nah, not my cup of tea.

[uncredited]

And then what?

Is anyone going to maintain all these Rust rewrites? Or are they going to die off like all those AI generated projects that are flooding everything now?

[EvaQ]

Re: And then what?

AFAIK it's closed source, and only on their servers aka cloud, which they are selling as a service. So ... maintainer is my first concern.

[Anonymous Coward]

Re: And then what?

https://github.com/tursodatabase

[volsano]

First year fun

The first year of any project is the fun one.

You start with no known problems in your design and implementation. None of your design goals have been ruled out as impractical. Very few bug reports have been logged.

You are in rainbow and unicorn land.

Many of us yearn to be back there.

To keep that new-coder energy, you have to start a new project from scratch every year or so.

Meanwhile, SQLite strides on, fixing bugs daily, even though it is one of the most tested software products in history - eg see https://www.sqlite.org/src/timeline?y=t

That sort of dedication is how you change the world.

[Gordon Shumway]

Re: First year fun

This.

The trick is not rewriting SQLite in $fad-of-the-week.

The trick is being there 25 years later.

[Neurons for Kryton]

This isn't a rewrite of SQLite — it's a Rust clone.

[Sproggit]

Confusing Caption

With an apology-in-advance for being a bit contrary... can I just observe that I found the title given to this article to be completely misleading?

It isn't the fact that Torso have now re-written a second database engine in rust that's significant here, but that their conversion/rewrite of SQLite prompted them to write a kind of "universal translator bytecode layer" that sits under their "Postgres API". That seems to be the "important bit" here - more significant than the choice of language.

I take on board - and agree with - all preceding comments that this risks being a dangerous approach - the point that a flaw in the core engine infects all "implementations" that sit on top of it being [in my view] the key take-away.

Or the fact that nowhere does the article explain why the developers chose to use rust for their project, when the original code for SQLite and Postgres is C++ (OK, with JAVA and Rust in the mix).

I've tried not to allow myself to get drawn in the direction of concluding "this is just dumb" and instead try and argue this from the other side, to think in terms of a "value proposition". If I were a developer or team lead and I went to my company/boss/board/other and said, "Hey! I'm going to spend a bunch of time and money rewriting various database implementations. I'm going to translate them from their native source language to Rust, and I'm going to remove their underlying data access engine and replace it with common bytecode - *that I derived from SQLite* ..." what would my sponsors say in response?

Somehow, I suspect that "Are you out of your ####ing mind?" might be in there somewhere.

More realistically, the only viable justification I could come up with was to help complex implementation environments. Say you have two applications: one written to Postgres and another written to e.g. MS SQL Server. Now say that you want your MS SQL application to have read-only access to your Postgres data, but that you further decided the way you want to implement this was to modify the code sitting on top of your MS SQL. In that admittedly contrived scenario it would make sense to be able to re-use all your MS-SQL objects and handlers, which means your app code will be leaner, simpler and easier to maintain. [Or you could run your abstraction layer there]. But that was the only excuse I could come up with for wanting to do this.

Are there others?

Maybe I'm getting cynical, but this struck me as a solution in search of a problem rather than anything that addresses a widely-seen problem.

Your mileage may vary.

[that one in the corner]

Re: Confusing Caption

> their conversion/rewrite of SQLite prompted them to write a kind of "universal translator bytecode layer" that sits under their "Postgres API". That seems to be the "important bit" here - more significant than the choice of language.

According to TFA, they created a VM for the db internals, the same way that SQLite already uses a VM for its internals. The new bit is they then had a go at writing a second front-end for that VM. Yes, that is different to what SQLite have done - but then the SQLite team don't have any particular need to try that trick. If re-using the Turso db VM like that does turn out to be a fully functional good idea (don't forget, by their own admission they don't yet have a 100% replacement for anything) maybe someone else will have a go at writing a new front-end to the SQLite VM, get some healthy competition going! Hey, maybe someone already has reused the SQLite code like that, just not trying to replicate a.n.other pre-existing db but to do something new; bound to have happened somewhere, the way programmers like to tinker, even if it isn't being bruited about the Internet.

> Maybe I'm getting cynical, but this struck me as a solution in search of a problem rather than anything that addresses a widely-seen problem.

Nah, I think you're probably right. I'm afraid I don't even see your contrived example as being usable, from what I've managed to glean about their project*: IIUC your example would rely on the MSSQL front-end being able to go through the VM and access the data in Postgres format (albeit read-only) and for the MSSQL front-end being complete enough to make use of all your home-grown handlers etc. But so far, their faux-SQLite's selling point is that it uses the same on-disc structure as SQLite. So following on from there, the Postgres-alike is either going to use the SQLite in-disc structure as well, OR it will replicate the Postgres on-disc. And similarly the touted faux-MSSQL. Neither of those directions would directly support the your MSSQL application reading your existing Postgres on-disc: it isn't possible (without a lot more info to back it up) to claim that, just because the faux-MSSQL and faux-Postgres would run on top of the same VM that they would automatically gain access to each other's on-disc data. Any more than it would be true if the whole thing was written in Java and happened to use *that* VM instead. Or Lua's VM. Or Python's.

* corrections, additions welcomed, as always.

[Christian Berger]

In a way it's how Free Software is supposed to be

You see something, you disagree with it, you decide to make something "better".

Now one can argue if their system will be any "better" or not, but that's the whole point. Code represents opinions and values. Different people can have different opinions. That is fine and a healthy way towards progress.

It is also the "Unix-Way". If you have small programs, you can just replace them individually. Thinking of a "better" cron which solves a problem you have? Why not write/fork your own version. Wanna try another programming language? All fine, and it'll just slot into the already existing system.

We have seen this with multiple implementations of tools like "runoff", "TeX" and the like.

How much trust we can put into those systems is, of course, a completely different question. In case of SQLite, the original already has deserved a _huge_ amount of trust, so that's a very high bar to pass... but there are people out there using MS-SQL or Excel as a database.

[Anonymous Coward]

Re: In a way it's how Free Software is supposed to be

Indeed. Better is subjective and what might be a step forward to one person could be a step backwards for another. It's great that people are always trying to dethrone the current best solution with something new - at worst they've probably grown as engineers and at best they've moved the state-of-the-art forward a notch. Persuading people to migrate from Postgres is going to be an especially tough sell though.

[Daniel Pfeiffer]

Re: In a way it's how Free Software is supposed to be

Speaking of "TeX" (and "LaTeX",) they have AFAIK not been rewritten in Rust. Instead there is "typst" a rather easier and more modern alternative.

[that one in the corner]

Re: In a way it's how Free Software is supposed to be

What? Nobody has written a TANGLE2Rust yet? Quick, to Claude Code*

Ha, bet they haven't even done a RustWEB and used it to write proper literate Rust code. Nah, mixing in Markdown doesn't really cut the mustard.

* Oh blast, I don't a subscription; hey ho, probably all for the best.