manatee

This would be the repositories themselves and cookie-session storage. Imagine repos have an incrementing id. Then if you have two masters a repo gets created with the same id on both masters. This is very bad because now two people can see each others data. You have the same problem with cookies. Imagine if you have user id as an incrementing id. The two masters create a user with the same id and an encrypted cookie or another storage system (redis) stores the user id. Now depending on which master you get routed to you appear as a different user. Weirdly enough this would always be a problem based on how their failover system works. The only safe way i know how to turn a master-slave system into a safe HA system is how joyent does it (https://github.com/joyent/manatee/blob/master/docs/user-guid... : you basically need a vote at commit time. if you have this then only 1 master can commit). However, I'm guessing most of the time they only have two masters running for < 1 minute but probably this time they had two masters running for a long time. EDIT: oh. they said it affected issues & pull requests [they have different dbs for different stuff] so repo and authentication stuff wouldn't apply. oh, they said no data was lost as well so presumably that would exclude a split brain running for a long period of time.

nchronous slave basically adjudicates because it can only be promoted once to slave either by the master or the original slave. basically writes are only accepted if at least 2 nodes acknowledge the write and the system can survive one node dying. but because you need co-operation of 2 of 3 nodes in order to actually commit data in the system it doesn't have the split-brain issues you have with an external system checking for health and then telling nodes to stop connecting to the unhealthy master. i think the actual state machine used is a bit more complicated because they need to support removing/adding nodes and i think they end up requiring zookeeper in order to coordinate. flynn [https://flynn.io/docs/databases] also have a HA solution for postgres/mysql based on this. though, i can see why maybe this kind of setup might not perform as well as one where you are accepting writes after committing 2 out of N to the read replicas. especially when you already need lots of read replicas.