No comments yet.
- Tansu is an Kafka compatible broker with S3 or PostgreSQL storage - Broker schema validation of Proto/Avro/JSON backed topics - Proto/Avro/JSON backed topics are also written as Parquet record batches for easy ingestion by downstream tooling
Step by step tutorial: https://blog.tansu.io/articles/parquet
Tansu is written in async Rust with the Tokio runtime. We build a single static linked binary that contains the broker, proxy and admin/utility commands in ~60MB. Also available as a from scratch docker container.
The Kafka protocol is implemented with serde using proc macros to transform the JSON definitions into Rust data types (https://blog.tansu.io/articles/serde-kafka-protocol).
Proto/Avro/JSON is converted into Apache Arrow data types, and then into Apache Parquet with a sprinkle of Apache Datafusion.
We also have an example of it running on fly.io with tigris (https://blog.tansu.io/articles/fly-with-tigris) that scales brokers back to 0 between requests.
Looking forward to any feedback and hope that Tansu can improve your workflow!
