cruise-control

See my previous answer further up in this sub-thread. Neither Kafka nor BookKeeper require data repartitioning when adding new nodes. Instead, both require data rebalancing, which moves some data from existing nodes to the newly added nodes. Think: repartitioning changes the logical layout of the data, which can impact app semantics depending on your application; whereas data (re)balancing just shuffles around stored bytes "as is" behind the scenes, without changing the data itself. The confusion stems probably from the two words sounding very similar. For Kafka, you use tools like Confluent's Auto Data Balancer (https://docs.confluent.io/current/kafka/rebalancer/index.htm...) or LinkedIn's Cruise Control (https://github.com/linkedin/cruise-control) that automatically rebalance the data in your Kafka cluster in the background. Pulsar has

"The rescue will begin in its own time" sounds like they deployed Cruise Control[1] to monitor their cluster. [1]: https://github.com/linkedin/cruise-control

We use acks=1 just fine. At least once is pretty okay for a lot of use cases. > In the case of pulsar ack quorum can be much smaller than replication factor so you don’t have to wait for slow replica. Also no number of node going down would cause new write to fail. And under replicated segment will fixed automatically using remaining healthy machines right away. I agree, all good points about Pulsar using BK, although worth noting the autorecovery daemon has to be explicitly enabled. But I'll mention Cruise Control again - it solves a all of the issues you've described without manual intervention. https://github.com/linkedin/cruise-control