maelstrom

Just earlier this month I was going through https://github.com/jepsen-io/maelstrom and following the demo implementing (a cut-corners version of) Raft, and I found it quite elucidating. The sample (ruby) code contains some bugs, and I had to use some understanding to fix them. (The bugs were of the kind where a dummy implementation from an earlier step isn't correctly changed)

at first i thought it was about https://github.com/jepsen-io/maelstrom/tree/main which shares the same name

Maelstrom [1], a workbench for learning distributed systems from the creator of Jepsen, includes a simple (model-checked) implementation of Raft and an excellent tutorial on implementing it. Raft is a simple algorithm, but as others have noted, the original paper includes many correctness details often brushed over in toy implementations. Furthermore, the fallibility of real-world hardware (handling memory/disk corruption and grey failures), the requirements of real-world systems with tight latency SLAs, and a need for things like flexible quorum/dynamic cluster membership make implementing it for production a long and daunting task. The commit history of etcd and hashicorp/raft, likely the two most battle-tested open source implementations of raft that still surface correctness bugs on the regular tell you all you need to know. The tiger

I'd add Gossip Glomers [0] which are pretty similar to jepsen-io/maelstrom Documentation [1]. [0] <a href="https://fly.io/dist-sys"> Gossip Glomers</a> [1] <a href="https://github.com/jepsen-io/maelstrom#documentation"> jepsen-io/maelstrom Documentation</a>

Love it. Thanks for putting this together! The actual challenges here [0]. Though I'm curious: are these different from the chapters in the Maelstrom documentation [1]? There seems to be a bit of overlap anyway. [0] https://fly.io/dist-sys/ [1] https://github.com/jepsen-io/maelstrom#documentation

I've actually written a little mini-Jepsen workbench for folks who want to practice implementing a distributed key-value store. Might be worth a spin! https://github.com/jepsen-io/maelstrom

Kinda depends on what you're doing, how much of Jepsen you're using, and how complex the system-specific code is. You can write a minimal Jepsen test in ~100 lines of code, if that's helpful. Jepsen and its main supporting libraries (Elle and Knossos) clock in at about 19K lines of code; a little over six years of full-time work. For simulation testing, I'd suggest looking at Maelstrom, which uses Jepsen to provide a sort of workbench for writing toy Raft implementations in any language. You give it a binary which takes messages as JSON on STDIN and emits messages to STDOUT; it spawns a bunch of "nodes" (local processes) of that binary, connects them via a simulated network, generates pathological network behavior, simulates client requests, and verifies the resulting histories with Jepsen. https://github.com/jepsen-io/maelstrom