streamparse

One of the nice things about Storm, in particular, is that the creator of the project thought about multiple language support from the start. We use Storm via our open source project, streamparse[1], and this enables usage of normal Python programs running under a Storm topology. In our case, we achieve multi-process parallelism and fault tolerance using Storm's simple multi-lang protocol, which is abstracted by our library. We also use Kafka natively from Python via the other module we released publicly, pykafka[2]. It provides support for nearly all of the Kafka protocol, and we even have an optimized C extension module in the works that may even be faster than the JVM consumer. My view on this is that infrastructure has to be implemented in some language. I don't lament the fact that Postgres was written in C or that Cassandra was writ

If you're interested in Python + Storm, we are making it happen via our open source project streamparse. See the project on Github[1] and also my PyCon US 2015 talk[2]. Your point about documentation is well-taken. We are trying to document streamparse + Storm usage from a Pythonista standpoint via our online documentation[3], e.g. here is our detailed Python API documentation[4]. [1]: https://github.com/Parsely/streamparse [2]: https://www.youtube.com/watch?v=ja4Qj9-l6WQ [3]: http://streamparse.readthedocs.org/en/stable/ [4]: http://streamparse.readthedocs.org/en/stable/api.html

Apache Storm[1] isn't exactly like Erlang/OTP, but it definitely aims to achieve similar goals. Though Storm is written in Java and Clojure, the infrastructure it provides is usable across languages thanks to something it calls the "multi-lang protocol". I work on a Storm library for Python that's called streamparse[2], and the goal of the project is to allow us to easily achieve Erlang OTP-style reliable processing atop open source infrastructure while still writing pure Python code. I also gave a PyCon talk about streamparse which you can find on YouTube[3]. It describes the motivation for the project -- which is to solve a large-scale real-time analytics problem with Python and do so in a reliable way, while beating Python's multi-core limitations (the GIL, etc.) [1]: https://storm.apache.org/ [2]: https://github.com/Parsely/streampars