jepsen

Jepsen Breaking distributed systems so you don't have to. Jepsen is a Clojure library. A test is a Clojure program which uses the Jepsen library to set up a distributed system, run a bunch of operations against that system, and verify that the history of those operations makes sense. Jepsen has been used to verify everything from eventually-consistent commutative databases to linearizable coordination systems to distributed task schedulers. It can also generate graphs of performance and availability, helping you characterize how a system responds to different faults. See jepsen.io for examples of the sorts of analyses you can carry out with Jepsen. Design Overview A Jepsen test runs as a Clojure program on a control node. That program uses SSH to log into a bunch of db nodes, where it sets up the distributed system you're going to test using the test's pluggable os and db. Once the system is running, the control node spins up a set of logically single-threaded processes, each with its own client for the distributed system. A generator generates new operations for each process to perform. Processes then apply those operations to the system using their clients. The start and end of each operation is recorded in a history. While performing operations, a special nemesis process introduces faults into the system--also scheduled by the generator. Finally, the DB and OS are torn down. Jepsen uses a checker to analyze the test's history for correctness, and to generate reports, graphs, etc. The test, history, analysis, and any supplementary results are written to the filesystem under store/<test-name>/<date>/ for later review. Symlinks to the latest results are maintained at each level for convenience. Documentation This tutorial walks you through writing a Jepsen test from scratch. An independent translation is available in Chinese. For reference, see the API documentation. What's Here provides an overview of Jepsen's namespaces and how they work together. Setting up a Jepsen Environment So, you've got a Jepsen test, and you'd like to run it! Or maybe you'd like to start learning how to write tests. You've got several options: AWS One easy way to get a Jepsen cluster is using Jepsen's AWS Marketplace listing. You can either run a single EC2 VM with serveral DB nodes in containers, or a cluster of separate EC2 VMs---this is a little more expensive, but significantly faster, and lets you test clock skew. The Marketplace VMs come with an hourly fee (generally $1/hour/node), which helps fund Jepsen development. Once you've subscribed to the AWS Marketplace listing, go to your AWS Marketplace Subscriptions list, click Jepsen, then "Launch new instance". For a real cluster of EC2 VMs, pick "AWS Cloudformation" and click "Launch with Cloudformation". At the "Create Stack" page, click "Next". On the "Specify Stack Details" page, give the stack a name like jepsen-test. Pick the SSH key you'd like to use to log in. Enter the CIDR IP range you'd like to give access to the control node; you can use your own (public) IP address, or 0.0.0.0/0 if you'd like access from anywhere. Click "Next". On the "Configure stack options" page, click the checkbox at the bottom to acknowledge that the cluster will create some IAM resources (a tiny S3 bucket, a few security groups, and the cluster of EC2 VMs, all torn down automatically on termination). Click "Next". On the "Review and create" page, click "Submit". On the Cloudformation stack page, a timeline will appear as CloudFormation sets up the cluster---this will take about four minutes. Click the "Outputs" tab to get the control node's public DNS address---something like ec2-12-3-45-67.compute-1.amazonaws.com. SSH to the control node using ssh -i <your-ec2-ssh-key.pem> admin@ec2-<whatever>.compute-1.amazonaws.com. Your DB nodes are named n1, n2, and you can usually run a test like so: lein run test --username admin --nodes-file ~/nodes ... For a single VM with DB containers, pick "Amazon EC2", and "Launch from EC2 Console". Choose version 0.3.11 and click "Launch from EC2". Pick your favorite instance type (an m8i.xlarge should work fine), set the root volume storage to something reasonable--say 32 GB, and click "Launch Instance". Click the instance ID in the green notification that pops up, and copy the "Public DNS" address of the instance---something like ec2-12-3-45-67.compute-1.amazonaws.com. SSH to the control node using ssh -i <your-ec2-ssh-key.pem> admin@ec2-<whatever>.compute-1.amazonaws.com. Launch the DB containers by running jepsen-containers-start. Your DB nodes are named c1, c2, ..., and are listed in ~/container-nodes. You can usually run a test like so: lein run test --nodes-file ~/container-nodes ... LXC You can set up your DB nodes as LXC containers, and use your local machine as the control node. See the LXC documentation for guidelines. This might be the easiest setup for hacking on tests: you'll be able to edit source code, run profilers, etc on the local node. Containers don't have real clocks, so you generally can't use them to test clock skew. VMs, Real Hardware, etc. You should be able to run Jepsen against almost any machines which have: A TCP network An SSH server Sudo or root access Each DB node should be accessible from the control node via SSH: you need to be able to run ssh myuser@some-node, and get a shell. By default, DB nodes are named n1, n2, n3, n4, and n5, but that (along with SSH username, password, identity files, etc) is all definable in your test, or at the CLI. The account you use on those boxes needs sudo access to set up DBs, control firewalls, etc. BE ADVISED: tests may mess with clocks, add apt repos, run killall -9 on processes, and generally break things, so you shouldn't, you know, point Jepsen at your prod machines unless you like to live dangerously, or you wrote the test and know exactly what it's doing. NOTE: Most Jepsen tests are written with more specific requirements in mind---like running on Debian, using ip