solace-nisshi

And the enshittification treadmill continues. Great time to be a kafka alternative. I'll start. https://github.com/tansu-io/tansu

For Rust-based Kafka alternatives, I like Tansu[1]. It at least provides Kafka API parity, and critically also gives users a pluggable backend (embedded SQLite, S3 for low cost diskless type workloads and Postgres because just use Postgres) It’s nice to try and out innovate Kafka, but I fear the network effect can’t be beaten unless the alternative is 10x better. Something like Warpstream’s architecture[2] had a shot at dethroning Kafka, but critically even they adopted the Kafka API. Sure enough, Apache Kafka introduced a competing feature[3] within two years of warpstreams launch too. [1] - https://github.com/tansu-io/tansu [2] - https://www.warpstream.com/ [3] - https://topicpartition.io/blog/kip-1150-diskless-topics-in-a...

These are real trade offs! Client impl. can be re-created once in a library/extension and be done with. The network effect of Kafka is the hardest to topple. The API is standard. If you need that connectivity, then use Kafka. An alternative idea I did not mention is to use a Kafka API proxy on top of Postgres. Tansu is developing this (amongst other pluggable backends) - https://github.com/tansu-io/tansu; That solution would retain both the client implementations and tools (network effect). But it comes at extra complexity

What is it? - Tansu is an Kafka compatible broker with S3 or PostgreSQL storage - Broker schema validation of Proto/Avro/JSON backed topics - Proto/Avro/JSON backed topics are also written as Parquet record batches for easy ingestion by downstream tooling Step by step tutorial: https://blog.tansu.io/articles/parquet Tansu is written in async Rust with the Tokio runtime. We build a single static linked binary that contains the broker, proxy and admin/utility commands in ~60MB. Also available as a from scratch docker container. The Kafka protocol is implemented with serde using proc macros to transform the JSON definitions into Rust data types (https://blog.tansu.io/articles/serde-kafka-protocol). Proto/Avro/JSON is converted into Apache Arrow data types, and then into Apache Parquet with a sprinkle of Apache Datafusion. We also have an exa