synapse

Has anyone actually done this at scale and seen how it works? We've talked about it a few times because Consul looks really cool but it doesn't appear to have the full capabilities of SmartStack yet, in particular Synapse (https://github.com/airbnb/synapse) for intelligently doing stats socket updates, intelligent reloads, or controlling how HAProxy reloads over time. Also consul only has two levels of services right, "local" and "remote"? I'm not sure that is sufficiently powerful to describe SOA's which need to be able to differentiate services on the machine, rack, colo, datacenter, region locality? I suppose you could do what we do in SmartStack and just register the same service instance at a few different places since I think that consul can register arbitrary services. Maybe all we need is a consul watcher in Synapse, and a consul-

The most significant issue with HAProxy and large SOAs is the lack of really good dynamic re-configuration. You don't want to have to manually change your configuration every time someone wants to add or change load balancing to a service, and if you do it automatically you run into HAProxy reloads being a reliability problem as they drop traffic on the floor due to Linux being goofy in how it implements SYN handling. It looks like there is a lot of good advice on this thread for how to mitigate the configuration problem (bamboo, consul template, etc ...) but then you run into reloads, and even if you use zero downtime restarts as I described in my HAProxy deep dive (http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h...), you still end up with periodic ~50ms blips in your timings. We've found that Synapse (https://github.com/air

Airbnb recently merged some great changes into SmartStack [0, 1] provided by Yelp, among which are reducing the number of connections to zookeeper [2]. [0] https://github.com/airbnb/nerve/pull/71 [1] https://github.com/airbnb/synapse/pull/130 [2] https://github.com/Yelp/synapse/commit/82775562a35a89d60084f...