zerorpc-python

+1 on sticking to DEALER/ROUTER on top of a green-thread system like gevent. This is how zerorpc works [1], and it is the backbone of the dotCloud platform. Also note that most of the time PUB/SUB and PUSH/PULL are not a good idea either. The same results can be usually be achieved by returning a stream on top of ROUTER/DEALER (this is what zerorpc does). The performance gains of custom topologies are great in theory, but in a typical modern web or mobile stack, they are not worth the extra effort and lack of flexibility. The single best change we made to dotCloud's architecture was move away from custom topologies and stick to DEALER/ROUTER. [1] http://github.com/dotcloud/zerorpc-python

If you like this, you should check out ZeroRPC, it handles a lot of the boilerplate code you would need to write by hand. Links: - http://zerorpc.dotcloud.com - https://github.com/dotcloud/zerorpc-python

Zerorpc (http://github.com/dotcloud/zerorpc-python) uses msgpack as its primary serialization format. Among other things it is significantly more efficient for floating point data. At dotCloud we use it to move many millions of system metrics per day, so it adds up. Also worth noting: msgpack maps perfectly to json, so there's nothing keeping you from translating it to json on its way to the browser, where json is indeed a better fit. In practice this doesn't affect performance since you typically need to parse and alter the message at the browser boundary anyway, for some sort of access control.

Would live to see that integrated with zerorpc! http://github.com/dotcloud/zerorpc-python dotCloud has an internal tracing tool called zerotracer, but we don't yet have a visualization layer...

Ironically dotCloud started out in 2008 as a very similar project, with the same lofty goals. We had VM templating and versioning (https://github.com/dotcloud/cloudlets), cross-infrastructure deployment (https://bitbucket.org/dotcloud/vm2vm), various goodies based on container virtualization, copy-on-write filesystems (https://github.com/dotcloud/aufs.py), and an orchestration layer to tie it all together (I found an old fork at https://bitbucket.org/Foi3GraS/dotcloud). It powered various 3d-party "XaaS" throughout 2009 - here's a French provider that still seems to be alive: http://www.diva-cloud.com/softwares Then in 2010 we joined YC, and actually tested our claim that "you can make it publicly available as a service!". Within 6 months we had rewritten 100% of the codebase around the real constraints of deploying and being responsible

I'm curious about the constraints this would impose on alternate transports. How similar would they have to be to HTTP? For example, zerorpc (http://github.com/dotcloud/zerorpc-python) supports a request-response pattern (using ZMQ REQ/REP), and can return a stream as a response. That maps nicely to http, and I could see that "mounted" as a mock http endpoint which is very interesting. But there are lots of features in zerorpc that I'm not sure how to map to an http library. For example: * zerorpc methods must expose positional arguments; how would that map to http query arguments? * zerorpc arguments can be arbitrary data structures. * zerorpc has no notion of headers. * etc. I'm excited at the idea of making cross-transport interop easier. At the same time I wonder if the mold of "HTTP-ness" might stifle the ability to think outside the

I hacked up a Faraday (my ruby HTTP client) plugin that uses ZeroMQ: https://github.com/technoweenie/faraday-zeromq. I honestly don't know if it's worth it though. It's easy enough to write your own ZeroMQ stuff. Example: https://github.com/dotcloud/zerorpc-python

There is a bug when using the edge-triggered fd from a zmq socket. I am not sure if it's fixed upstream yet or not. See here for an ugly workaround: https://github.com/dotcloud/zerorpc-python/blob/master/zeror...