pykafka

I just wanted to mention that pykafka is currently unmaintained and archived on GitHub: https://github.com/Parsely/pykafka pykafka was originally developed and maintained by my team at Parse.ly, but we no longer maintain it. We instead encourage folks to use confluent-kafka-python, which is what we have ourselves switched to in our production systems: https://github.com/confluentinc/confluent-kafka-python (pykafka was developed at a time before Confluent invested in their own Python binding. Some of the history of the project is described in this 2016 blog post[1] and our original 2015 announcement[2].) [1]: https://blog.parse.ly/pykafka-now/ [2]: https://blog.parse.ly/announcing-pykafka-python-support-for-...

If you're a Python user, I gave a talk at PyData called "Beating the GIL with Python" that covers all the tooling for parallel computing in Python, from the simplest (e.g. multiprocessing) to the more complex (Storm/Spark), with many tools in between, and what the trade-offs are. That may help you out. https://www.youtube.com/watch?v=gVBLF0ohcrE The rough summary is, if you need to do distributed multi-node computation with very low latency -- that is, end-to-end data processing time measured in millis or seconds -- then Storm will help you out. The "competitive" product in open source right now is pyspark + Spark Streaming. The benefit of Storm is that it is an older (more mature) project with many production deployments. My team at Parse.ly wrote the open source Python + Storm integration libraries, pystorm and streamparse, which help P

Sounds like old news, but if this is still an issue, PyKafka does allow balanced reads across a consumer group. https://github.com/parsely/pykafka

https://github.com/parsely/pykafka PyKafka is currently used in production at Parse.ly, and I've gotten feedback from a lot of other folks who are using it in production as well. The big benefit over kafka-python is that PyKafka supports multi-consumer groups that balance consumption via ZooKeeper with its BalancedConsumer interface. See this thread ( https://github.com/Parsely/pykafka/issues/334 ) for more detail on the differences between the two libraries. The PyKafka project is prioritizing support for Kafka 0.9 in the next few weeks/months. This includes ensuring that the existing consumers work against the updates to the 0.8.2 consumer API as well as implementing support for the new consumer API introduced in 0.9. Roadmap information can be found here ( https://github.com/Parsely/pykafka/blob/master/doc/roadmap.r... ).

Yes, each Python Bolt and Spout runs as a separate process. Though, by changing the paralellism hint, you change the number of Python processes spawned per component (e.g. :p 8 = 8 Python processes running the same bolt code). This is because, under the hood, it all uses ShellBolt/ShellSpout in the Storm layer, which is all process-based. But I actually find this to be the "purer" way to run Storm, anyway. Why deal with threads when you don't need to. As Joe Armstrong, the creator of Erlang, once said (paraphrasing): "Processes are isolated environments for code where state can't be shared except through explicit messaging; threads are isolated environments where state is directly -- and dangerously -- shared. Why would you want threads when you could have processes?" Ofc, I realize, there are times when threads' lightweightness matters,

One of the nice things about Storm, in particular, is that the creator of the project thought about multiple language support from the start. We use Storm via our open source project, streamparse[1], and this enables usage of normal Python programs running under a Storm topology. In our case, we achieve multi-process parallelism and fault tolerance using Storm's simple multi-lang protocol, which is abstracted by our library. We also use Kafka natively from Python via the other module we released publicly, pykafka[2]. It provides support for nearly all of the Kafka protocol, and we even have an optimized C extension module in the works that may even be faster than the JVM consumer. My view on this is that infrastructure has to be implemented in some language. I don't lament the fact that Postgres was written in C or that Cassandra was writ