raft

Now there's three of them https://github.com/hashicorp/raft

Anyone else came thinking to find something on the Raft consensus library[1]? [1] https://github.com/hashicorp/raft

> Our Solution: Use Redis and Postgres Database Polling + leader election using RedLock To me this seems one of the worst architectural choices they could have made. I'm assuming they had some constraints that were not mentioned in the article to justify this decision. But for a simple distributed circuit breaker this seems far from a reasonable solution. Besides that, there is one implicit assumption that was made that I'm not sure is true. By having a 'leader' you are basically saying that your network connections never fail or that they always fail at the same time. Sure, under normal conditions this assumption doesn't seem that bad. But as soon as things start to degrade you may have a pretty bad outage in your hands. If I had to implement this feature I would probably use raft[0] to communicate between pods. And would also keep track

>You could just use sqlite with :memory: for the Raft FSM That's the basic design that rqlite[1] had for its first ~7 years. :-) But rqlite moved to on-disk SQLite, since with WAL mode, and with 'PRAGMA synchronous=OFF' [2], it is about as fast as writing to RAM. Or at least close enough, and I avoid all the limitations that come with :memory: SQLite databases (max size of 2GB being one). I should have just used on-disk mode from the start, but only now know better. (I'm guessing you may know some of this because rqlite uses the same Raft library [3] as Nomad.) As for the upgrade issue you mention, yes, it's real. Do you find it in the field much with Nomad? I've managed to introduce new Raft Entry types very infrequently during rqlite's 10-years of development, only once did someone hit it in the field with rqlite. Of course, one way to

Agreed, in the sense that while rqlite has a lot in common with etcd (and Consul too -- Consul and rqlite share the same Raft implementation[1]) rqlite's primary use case is not about making it easy to build other distributed systems on top of it. [1] https://github.com/hashicorp/raft

FWIW, I built hraftd[1] many years ago to make it easy to play with a simple distributed system, but one that uses a production-grade implementation of Raft[2]. You can spin up a cluster in seconds on a single machine, kill nodes, watch a new Leader get elected, and so on. It's written in Go, so it'll help if you are familiar with Go. But the code is not difficult to understand even if you don't. [1] https://github.com/otoolep/hraftd [2] https://github.com/hashicorp/raft

I have found the performance tests very tricky to get to pass without having any input from others. The assignment is really very unforgiving, I would wager the test suite is comparable to how commercial Raft implementations are tested (e.g. https://github.com/hashicorp/raft)

Yup! etcd [0] and hashicorp [1] have Raft libraries for Go. I've written about using the hashicorp one before [2, 3]. [0] https://pkg.go.dev/go.etcd.io/etcd/raft/v3 [1] https://github.com/hashicorp/raft [2] https://notes.eatonphil.com/distributed-postgres.html [3] https://notes.eatonphil.com/minimal-key-value-store-with-has...

Something I'm not clear on from reading the post - the documentation for the Apply() method on the Raft type (https://pkg.go.dev/github.com/hashicorp/raft#Raft.Apply) mentions that "This must be run on the leader or it will fail". With the code as written, it seems like a user running psql has to know beforehand which process is the Raft leader for running CREATE TABLE/INSERT statements, and there's no way to redirect those messages to the Raft leader. Is this something that could be handled (and was just left out for brevity's sake), or is there something fundamental I'm missing?

It's all handled by the Raft libraries (I know this because rqlite uses the same Raft library at this blog post). When integrating with Hashicorp's Raft code, your code must implement Apply()[1]. The Hashicorp code promises not to call Apply() until a quorum of nodes have been persisted the data to the Raft log. That way when your code is called to modify your state machine (FSM), quorum has already been achieved by the Raft system. Architecturally (in rqlite's case) a node looks something like this: https://docs.google.com/presentation/d/1Q8lQgCaODlecHa2hS-Oe... [1] https://pkg.go.dev/github.com/hashicorp/raft#FSM

Hey, you could inspire yourself by hashicorps raft implementation written in go and build one for typescript. Code is quite good to read and Go ins't that far away from typescript. https://github.com/hashicorp/raft

In general HashiCorp's repos are high quality: https://github.com/hashicorp/raft Example application: https://github.com/Jille/raft-grpc-example

actually it's using "github.com/hashicorp/raft" and not its own version. So since Hashicorp survived Aphyr test, it should be fine.

Raft is a large improvement over Paxos for practical implementations. But it's still tricky to get right. As far as I know, the only widely used, battle tested Raft implementation is github.com/hashicorp/raft. Which is why so many distributed systems are being built on golang over the last few years. I don't know if there is any Java raft implementation which has reached that level of maturity yet - but it seems like Confluence is trying with kraft.

This is the difference between a library and a framework/project. For example, there's a great Raft library for Go [1] that any project can use to implement distributed consensus without a separate running program. I find this to be a better approach with the same collective development and community testing advantages but without requiring more operational overhead. 1. https://github.com/hashicorp/raft