The core idea here is: let SQLite run next to your application on the server; but have all features a client-server database give you.
What's the spec for this?
- SQLite runs next to server as production database
- That way, reads and writes are very fast
- In dev, some sort of worker auto-copies the prod DB to the local repo. Production bugs can be reproduced easily and code fixed quickly
- Have an interface that lets you access, view, and modify data in the production DB, kind of like Firebase. Might need a server of its own... or couldn't that just be the app server itself?
- SQLite auto-backs up to a bucket (like Litestream)
I kinda really want this, but haven't found anything quite like it. I've seen Turso, but it seems they focus more on global replication instead of the OSS developer experience I'm looking for.
What do you think? What am I missing?
> - SQLite runs next to server as production database
Embedded Replicas[1]
> - In dev, some sort of worker auto-copies the prod DB to the local repo. Production bugs can be reproduced easily and code fixed quickly
Maybe not exactly this, but you can replicate a DB from another with the CLI tool. Then just use that as a dev db.
> - Have an interface that lets you access, view, and modify data in the production DB, kind of like Firebase. Might need a server of its own... or couldn't that just be the app server itself?
I don't use their web interface, but I think it does allow this? I don't know for sure though.
> - SQLite auto-backs up to a bucket (like Litestream)
Turso does have point-in-time recovery[2]
[1] https://docs.turso.tech/features/embedded-replicas/introduct... [2] https://docs.turso.tech/features/point-in-time-recovery
What is Mvsqlite? According to the author it's a distributed, MVCC SQLite that runs on top of FoundationDB.
https://github.com/V-Sekai/mvsqlite
Made an Elixir client, a Godot Engine client and CLI.
Supabase on Mvsqlite would be great! I am still puzzling over Mvsqlite's write performance, but its read performance makes sense to me.
# ycsb (workloadf, 10000, --wire-zstd)
Run finished, takes 1m18.086881433s
READ - Takes(s): 78.1, Count: 99957, OPS: 1280.3, Avg(us): 27061, Min(us): 1409, Max(us): 98751, 99th(us): 59391, 99.9th(us): 81791, 99.99th(us): 94591
READ_MODIFY_WRITE - Takes(s): 78.0, Count: 50061, OPS: 641.6, Avg(us): 69636, Min(us): 13320, Max(us): 205823, 99th(us): 121215, 99.9th(us): 189823, 99.99th(us): 203007
UPDATE - Takes(s): 78.0, Count: 50095, OPS: 642.0, Avg(us): 42630, Min(us): 7792, Max(us): 200319, 99th(us): 81535, 99.9th(us): 175487, 99.99th(us): 195199
From the github actions tests the original author wrote.
I gotta say, your idea of a Supabase-like system for SQLite got me thinking. It's an interesting concept. SQLite running next to your server could certainly speed things up, but don't forget about potential issues with concurrent writes. SQLite might not be as graceful as other databases in handling those.
The dev-prod synchronization you mentioned is a neat idea. Just remember to keep things secure and don't expose sensitive data. Also, think about the size of your production database. If it's too big, this might not be the best way to go.
Having an interface to access, view, and modify data straight from the production DB would be super useful. Depending on how complex this gets, you could either make it part of the application server or set up a separate server.
Auto-backup to a bucket is definitely a thumbs up from me. Tools like Litestream can be a lifesaver here.
Have you heard of Datasette? It's an open-source tool that lets you explore and publish data from SQLite databases. It could be a good starting point for what you're trying to build.
Your idea sounds like it could really shake things up. Keep us posted on how it goes. Good luck!
Anyway, there is a new generation of database-systems/frontends, like Grist, NocoDB, Airtable, etc. But they are more focused on a user-friendly all round experience, not so much a good experience. But some of them do support working with SQLite.
Could be good, you need to rethink all the distributed DB stuff.
Does a node deal with certain subset (e.g. certain customers) so you use customerid as a partition key.
CRDTs?
Once a node fills up to say x% capacity do you split that out (using consistent hashing to minimize effect of such spits).
You probably end up reinventing one of the distributed DBs once you make it production ready.
I think it is a lot of work unless you have simplifying requirements e.g. just give each customer a seperate VM and then use a mounted cloud storage to scale up space to some max like 10Tb then maybe cap it there. Write your db log to s3 glacier for a backup.
Second replica for failover.
If you are asking in human terms, its instantaneous to start, it simply creates a few files etc and starts a process. If you were using Sqlite, there is no extra associated process. But the processes are simply waiting for your input, so normally this is not really extra work.
In benchmark terms, the way I'm using it startup time is not a huge issue, instead being able to use postgres is helpful (has extensions and you know you can eventually move to a hosted postgres if you want)
If startup really matters, the startup can be amortized (if multiple processes want to access the same file, they get a handle to the same already started process, so only the first one would have waited for startup).
In terms of ongoing resource consumption, I can easily have a bunch of separate servers running for different files (it depends on your workload how much work will happen behind the scenes.) I'm not sure how much extra work postgres needs to do vs sqlite beyond what is kind of inherent for a workload (eg index creation, saving a bunch of data etc).
- For our hosted version, we use Litestream; we lack a UI for accessing data.
The minute you put it behind a socket, you lose that benefit. Also you would have to implement an efficient wire protocol.
I just don't see the added value here.
