shuttle

My current thinking on model checking (still evolving): Modeling languages are useful to check the correctness of an algorithm during development. During development, a model can also serve as a specification for the actual implementation. This requires that your the modeling language is readable to a broad range of developers, which TLA+ is not. We have been experimenting with FizzBee (fizzbee.io) which looks promising in this regards. When you go to prod, you really want to test your actual implementation, not a model of it. For this you want something like https://github.com/awslabs/shuttle (for Rust), or https://github.com/cmu-pasta/fray (for Java). Or use something custom.

Benches look promising! My main concern is validating correctness; implementing good concurrency primitives is always challenging. Have you looked into testing against a purpose-built concurrency model checker like tokio-rs/loom [1] or awslabs/shuttle [2]? IMO that would go a long way towards building trust in this impl. [1] https://github.com/tokio-rs/loom [2] https://github.com/awslabs/shuttle

If you happen to be coding in Rust, for really robust concurrency testing, I cannot recommend enough the AWS Shuttle library (https://github.com/awslabs/shuttle) which can find insanely complicated race conditions. What the Shuttle library is doing is basically automatically going through all the permutations of the schedule diagrams described int his blog post. We used it at AWS to verify the custom filesystem we wrote to power AWS S3. If you're curious, I wrote a little tutorial on it here: https://grantslatton.com/shuttle

Very cool! I implemented some shared-memory atomic snapshots in Rust [0] and also did my best to take automated testing very seriously. I started out using loom [1], the library mentioned in the article, but latter switched to shuttle [2]. Shuttle's approach is to be randomized, instead of exhaustive like loom. However, the scheduler does still give probabilistic guarantees about finding bugs. I found that shuttle was faster and scaled to more complicated test scenarios. Similar to the article, shuttle also lets you save the random seed if a particular schedule causes the test suite to fail. Being able to quickly reproduce failing tests is really important and enables you write explicit test cases for bugs that were previously caught and fixed [3]. [0] https://github.com/kaymanb/todc/tree/main/todc-mem [1] https://github.com/tokio-rs/loom

I do a lot of manual proptesting in Rust that all look something like: let mut rng = rand::thread_rng(); for action_count in 1..4 { for _ in 0..10_000 { let seed = rng.gen::<u64>(); eprintln!("let seed = {seed};"); let mut rng = ChaChaRng::seed_from_u64(seed); i.e. top level true randomness, then a bunch of nested loops (only 2 here, but some tests have more) to go from low-complexity cases to high-complexity then generate a seed to seed a deterministic PRNG, and print it out so if the test fails, I just copy and paste the error seed to replay the error case I have found doing this manual proptesting to be faster, more flexible, and generally less fuss than using any frameworks or libraries That said, for really robust concurrency testing, I cannot recommend enough the AWS Shuttle libra

This is what we did for our KV store at S3: https://www.amazon.science/publications/using-lightweight-fo... Using https://github.com/awslabs/shuttle which works on our real Rust code.

They've also developed a library that plugs into their actual Rust code to verify it (rather than writing a secondary model in TLA+ or P, it's easier to verify the actual system source code). See https://github.com/awslabs/shuttle and a whitepaper at https://www.amazon.science/publications/using-lightweight-fo... Disclaimer: used to work at AWS and had some involvement in this stuff

Loom is really awesome, though it is focused on exhaustive testing, so not suitable for code that has a lot of possible interleavings (e.g. due to a ton of threads, or a large body of code). There is a new project out of AWS called Shuttle [1] which is like Loom, but it does random exploration instead of exhaustive exploration, which enables massively distributed testing of really complicated stuff. [1] https://github.com/awslabs/shuttle