And one of the first patterns I wanted to use was this. Just a read-only event log that's replicated, that is very easy to understand and operate. Kafka is a beast to manage and run. We picked it at my last company -- and it was a mistake, when a simple DB would have sufficed.
https://github.com/simonw/sqlite-utils https://litestream.io/
I think part of my issue is that a lot of uses of it end up having a big global lock on the database file (see: older versions of Emby/Jellyfin) so you can't use it with multiple threads or processes, but I also haven't really ever find a case to use it over other options. I've never really felt the need to do anything like a JOIN or a UNION when doing local configurations, and for anything more complicated than a local configuration, I likely have access to Postgres or something. I mean, the executable for Postgres is only ten megs or twenty on Linux, so it's not even that much bigger than SQLite for modern computers.
PRAGMA journal_mode = WAL;
One note on the backup/migrate, I think you need a shared lock on the database before you copy the database. If you dont, the database can corrupt. SQLite docs have other recommendations too:
> kafka compatible
Kafka is not a straightforward protocol and has a few odd niches. Not to mention that message formats have changed over the years. Even the base product has recently dropped support for some of the oldest API versions. And there are still plenty of clients out there using old versions of librdkafka (he says from experience).
So I'd be interested how (backward-)compatible they are.
Some of the detail: https://blog.tansu.io/articles/serde-kafka-protocol
However, there are definitely cases I am sure where Tansu isn't compatible. For example, Kafka UI (kafbat) reports a strange error when doing a fetch (despite actually showing the fetched data), which I've yet to get to the bottom of.
If you find any compatibility issues, then please raise an issue, and I can take a look.
Also, are there any benchmark on how Tansu with S3 storage would perform in comparison to Kafka or something like WarpStream?
https://blog.tansu.io/articles/route-layer-service
My itch for SQLite was smaller scale (and reproducible) environments, e.g., development, test/integration (with a single file to reset the environment). PostgreSQL was intended for "larger scale", with (database level) partitioning of Kafka records on each topic/partition, and replication for leader/follower setups, which might work better for HA. S3 for environments where latency is less of any issue (though with the SlateDB/S3 engine that might change).
S3: Not yet. I've been working through tuning each engine, S3 is next on the list.
same for redis, kafka, ...