HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
airbnb
synapse
Ruby2013–2020
airbnb / synapse

synapse

A transparent service discovery framework for connecting an SOA

Explore on GitHubRead Hacker News discussions
9Distinct people
16Mentions
1Explicit recommenders
2.1KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

A transparent service discovery framework for connecting an SOA

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

Do health checks work within a VPC? My understanding was they don't, so this only works for externally facing services. I agree Haproxy is solid, but ELBs are wonderful for…

toomuchtodo used a recommendation phrase. One indexed comment matches. This is a phrase match, not a verdict.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year16 dated mentions

Observed mentions: 2013-10-23 22:21:50 UTC to 2020-03-05 19:27:39 UTC.

02462013-01-01 00:00:00 UTC to 2014-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments20132014-01-01 00:00:00 UTC to 2015-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2015-01-01 00:00:00 UTC to 2016-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 6 mentions; 0 submissions, 6 comments20162017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2020

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20132013-01-01 00:00:00 UTC2014-01-01 00:00:00 UTC101
20142014-01-01 00:00:00 UTC2015-01-01 00:00:00 UTC303
20152015-01-01 00:00:00 UTC2016-01-01 00:00:00 UTC303
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC606
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC202
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC000
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC000
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC101

What Hacker News says

16 sources
philsnow·Mar 5, 2020
> if nginx or haproxy could natively talk to Consul for upstream data, I’m not sure how much of this other stuff would have ever been necessary Airbnb's Smartstack works well for this. It's not built in to nginx as a module, but I think it's more composeable this way. Blog post: https://medium.com/airbnb-engineering/smartstack-service-dis... The two main components are nerve (health checking of services + writing a "I'm here and healthy" znode into zookeeper, https://github.com/airbnb/nerve) and synapse (subscribes to subtrees in zookeeper, updates nginx/haproxy/whatever configs with backend changes, and gracefully restarts the proxy, https://github.com/airbnb/synapse). It's fairly pluggable too if you don't want to use haproxy/nginx.
In “Let’s use Kubernetes.” Now you have eight problems
tomfitz·Nov 2, 2017
https://www.haproxy.com/blog/dns-service-discovery-haproxy/ (which is a link on the OP) says TTL on SRV records "is ignored by HAProxy as HAProxy maintains its own expiry data which is defined in the configuration". Is this true for A records too? If so, neither haproxy nor nginx expire cached A records. Nginx Plus does, and a few nginx plugins do, however. https://github.com/airbnb/synapse is a process that polls DNS, and updates haproxy config accordingly and SIGHUPs haproxy I've used synapse to solve this issue, but it's a moving piece I'd rather not have involved.
In HAProxy 1.8
jolynch·May 5, 2017
Disclaimer: I help maintain Synapse. I highly recommend this tool. Yelp has used it in production for years to manage a fairly large PaaS (hundreds of services, thousands of containers, constant churn); it's proven quite flexible and resilient. Synapse is available on github [0], and we've open sourced our automation used to create a highly available service router using Synapse as well [1][2]. [0] https://github.com/airbnb/synapse [1] https://github.com/Yelp/synapse-tools/tree/master/src/synaps... [2] http://paasta.readthedocs.io/en/latest/yelpsoa_configs.html#...
In Truly Seamless Reloads with HAProxy
jolynch·Dec 1, 2016
Funny enough we're doing something pretty similar for our external lb tier but instead of a custom proxy we're just using nginx proxying back to HAProxy. Your solution is really nifty though because there aren't two accepts going on. In the thing that we're trying nginx terminates TCP (and SSL if needed), proxies back to a unix socket where HAProxy is listening. This gives us the best of both worlds: we can reload HAProxy all day long and socket listeners are hitless, and we rarely need to reload the nginx config which comes out of the box with hitless TCP sockets. But it's not quite as cool as this solution because HAProxy and nginx both have to accept. We haven't decided that we're going to do our internal load balancing this way yet because the listening ports change so often, but if we do I'll probably be working on https://github.com
Read the rest
/airbnb/synapse/pull/203 to do it. The PR makes it so that Synapse can manage both nginx and HAProxy simultaneously (to deal with the extra system complexity of an additional proxy). Or we'll try this cool thing out :-P
In GitHub Load balancer: HAProxy zero-downtime, zero-delay reloads with multibinder
jolynch·Nov 1, 2016
(I help maintain SmartStack) I think it's really interesting that "what we've already got setup" is such a big driver in which systems we pick. For example, in 2013 Yelp already had hardened Zookeeper setups and Consul didn't exist ... and when it did exist Consul was the new "oh gosh they implemented their own consensus protocol" kid on the block, so we opted for what we felt was the safer option. I do have to be honest that I was also pretty worried about the ruby ZK library, but to be totally honest it's been relatively well behaved, aside from the whole sched_yield bug [1] occasionally causing Nerves to infinite loop shutting down. We fixed that with a heartbeat and a watchdog, so not too bad. Which technologies are available at which times really drives large technical choices like this. Consul template is undeniably useful, especial
Read the rest
ly when you start integrating it with other Hashicorp products like Vault for real time rolling your SSL creds on all your distributed HAProxies. And I think that the whole Hashicorp ecosystem together is a really powerful set of free off the shelf tools that are really easy to get going with. I do think, however, that Synapse does have some important benefits, specifically around managing dynamic HAProxy configs that have to run on every host in your infra. For example, Synapse can remove dead servers ASAP through the HAProxy stats socket after getting realtime ZK push notifications rather than relying on healthchecks (in production <~10s across the fleet, which is crucial because if HAProxy healthchecked every 2s we'd kill our backend services with healthcheck storms ... because we've totally done that ...), Synapse can try to remember old servers so that temporary flakes don't result in HAProxy reloads, and it can try to spread and jitter HAProxy restarts so that the healthcheck storms have less impact, all while having flexibility in the registration backend (Synapse supports any service registry that can implement the interface [2]). However, there are some pretty cool alternative proxies to HAProxy out there and one area that Consul is really doing well on is supporting arbitrary manifestations of service registration data using Consul template; SmartStack is still playing catch up there, supporting only HAProxy and json files (with arbitrary outputs on their way in [3]). I enjoyed the article, and thank you to the Stripe engineers for taking the time to share your production experiences! I'm excited to see folks talking about these kinds of real world production issues that you have to deal with to build reliable service discovery. [1] https://github.com/zk-ruby/zk/issues/50 [2] https://github.com/airbnb/synapse/blob/master/lib/synapse/se... [3] https://github.com/airbnb/synapse/pull/203
In Service discovery at Stripe
jolynch·Oct 26, 2015
Has anyone actually done this at scale and seen how it works? We've talked about it a few times because Consul looks really cool but it doesn't appear to have the full capabilities of SmartStack yet, in particular Synapse (https://github.com/airbnb/synapse) for intelligently doing stats socket updates, intelligent reloads, or controlling how HAProxy reloads over time. Also consul only has two levels of services right, "local" and "remote"? I'm not sure that is sufficiently powerful to describe SOA's which need to be able to differentiate services on the machine, rack, colo, datacenter, region locality? I suppose you could do what we do in SmartStack and just register the same service instance at a few different places since I think that consul can register arbitrary services. Maybe all we need is a consul watcher in Synapse, and a consul-
Read the rest
template for the Synapse configs, and we can get the best of all worlds. With Linux 4.4 SYN handling in Linux might even be fixed by Eric Dumazet being super hardcore (e.g. https://lwn.net/Articles/659199/) and we won't even have to worry about reloading HAProxy.
In HAproxy in the era of Microservices
jolynch·Oct 26, 2015
The most significant issue with HAProxy and large SOAs is the lack of really good dynamic re-configuration. You don't want to have to manually change your configuration every time someone wants to add or change load balancing to a service, and if you do it automatically you run into HAProxy reloads being a reliability problem as they drop traffic on the floor due to Linux being goofy in how it implements SYN handling. It looks like there is a lot of good advice on this thread for how to mitigate the configuration problem (bamboo, consul template, etc ...) but then you run into reloads, and even if you use zero downtime restarts as I described in my HAProxy deep dive (http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h...), you still end up with periodic ~50ms blips in your timings. We've found that Synapse (https://github.com/air
Read the rest
bnb/synapse) is best in class at managing HAProxy for us in a fairly large SOA. It is really intelligent about doing as much as possible dynamically over the HAProxy stats socket as opposed to reloading for every configuration change, and we can finely tune how often we are reloading HAProxy. The best part is that it is fully pluggable. Use DNS, marathon, zookeeper, etcd, etc...: Synapse can manifest that service registration information into HAProxy configurations.
In HAproxy in the era of Microservices
meatmanek·Aug 22, 2014
Consul is pretty neat, and in my experiments with it, it's really easy to spin up and modify a cluster. It seems like they really thought through the procedures. Another mature option for service discovery that this port doesn't mention is AirBnB's SmartStack: https://github.com/airbnb/synapse and https://github.com/airbnb/nerve. Here's a talk from DockerCon about using SmartStack and Docker: http://blog.docker.com/2014/07/dockercon-video-building-a-sm...
In Consul Service Discovery with Docker
igor47·Apr 18, 2014
i am constantly impressed at the hashicorp guys, who continue to release great tools. they actually released serf on the same day as we released nerve and synapse, which comprise airbnb's service registration and discovery platform, smartstack. see https://github.com/airbnb/nerve and https://github.com/airbnb/synapse that said, as i wrote my blog post on service discovery ( http://nerds.airbnb.com/smartstack-service-discovery-cloud/ ), dns does not make for the greatest interface to service discovery because many apps and libraries cache DNS looksups. an http interface might be safer, but then you have to build a connector for this into every one of your apps. i still feel that smartstack is a better approach because it is transparent. haproxy also provides us with great introspection for what's happening in the infrastructure -- who is t
Read the rest
alking to whom. we can analyse this both in our logs via logstash and in real-time using datadog's haproxy monitoring integration, and it's been invaluable. however, this definitely deserves a look if you're interested in, for instance, load-balancing UDP traffic
In Consul, a new tool for service discovery and configuration

Showing the 15 most recent of 16 sources. Enable JavaScript to search and browse every indexed mention.

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.