parking_lot

> These are OS primitives I'm talking about - I haven't checked out the standard library version but the parking_lot version uses a spinlock with thread sleep when the wait times get too high - it has no way of getting notified when the mutex gets unblocked nor does it support priority inversion. Uhh no, everyone in Linux userspace uses futexes these days to wait on a contended lock. https://github.com/Amanieu/parking_lot/blob/03d36d62fecbd85c...

> ParkingLot just uses pthread mutex and cond. That's interesting, I'm more familiar with the Rust parking-lot implementation, which uses futex on Linux [0]. > Sure that uses futex under the hood, but the point is, you use futexes on Linux because that’s just what Linux gives you It's a little more than that though, using a pthread_mutex or even thread.park() on the slow path is less efficient than using a futex directly. A futex lets you manage the atomic condition yourself, while generic parking utilities encode that state internally. A mutex implementation generally already has a built-in atomic condition with simpler state transitions for each thread in the queue, and so can avoid the additional overhead by making the futex call directly. [0]: https://github.com/Amanieu/parking_lot/blob/739d370a809878e4...

Fun post! An alternative to using futexes to store thread queues in kernel space is to store them yourself. E.g. the parking_lot[0] Rust crate, inspired by WebKit[1], uses only one byte to store the unlocked/locked/locked_contended state, and under contention uses the address of the byte to index into a global open-addressing hash table of thread queues. You look up the object's entry, lock said entry, add the thread to the queue, unlock it, and go to sleep. Because you know that there is at most one entry per thread, you can keep the load factor very low in order to keep the mutex fast and form the thread queue out of a linked list of thread-locals. Leaking the old hash on resizing helps make resizing safe. As a result, uncontended locks work the same as described in the blog post above; under contention, performance is similar to a fute

As of recently, macOS does support priority inheritance for a bunch of synchronization primitives [1], ranging from pthread mutexes to Mach IPC. But it's not perfect. For instance, read-write locks currently don't support priority inheritance, though they could. More problematically, when userland processes use any kind of custom synchronization scheme based on condition variables or similar, there's no way for the kernel to know which thread is expected to signal the condition variable and thus should inherit priority. That includes when libraries implement their own mutexes on top of that [2]... [1] https://opensource.apple.com/source/xnu/xnu-4903.241.1/osfmk... [2] https://github.com/Amanieu/parking_lot/issues/205

> Locking is a performance [...] destroying operation That is incorrect. Properly designed and implemented mutexes (like in the parking_lot crate [1]) are extremely fast in the uncontented case, and need just one byte of overhead (and no heap-allocated memory) per mutex. In the contented case, a lock-free algorithm still has to deal with arbitration between the different threads, so it's typically not faster than a lock-based approach. Lock-free algorithms do have certain advantages (like deadlock-freedom), but performance is typically not one of them. > Locking is a [...] scalability destroying operation Also incorrect. If you have N threads trying to append to a single queue, then you have a scalability problem whether you're using locked or lock-free data structures. The solution in such a case is fine-grained concurrency and a differe

> You're no longer guaranteed that cache changes propagate unasked. Huh? Yes, you can: Caches in multi-core ARM systems are still coherent, so once a write operation reaches any level of the cache hierarchy (typically level 1) and updates that cache line, all other cores will see the updated value when they read from that address. That's not what you need memory consistency operations, like load acquire and store release, for. You need those to make sure that a write from a write operation has been written to the cache, because it might still linger in the store buffer, which sits between the execution unit and the cache. > Rust has the advantage that its locks own data, so the compiler knows what has to be flushed. That's absolutely not how Rust works. The Rust compiler has no notion of a mutex, only of atomic operations. Mutexes are pur

> Beyond that I wish the article had explain a bit better why it chose these "better-than-std" crates. At least for parking_lot, its README has a long list with its advantages over std: https://github.com/Amanieu/parking_lot/blob/master/README.md

So this it acquires fully in userspace if there's no contention, it even spins if there is contention, and then if that wasn't enough it lets the thread sleep with the timeout. Which matches the description of that library: https://github.com/Amanieu/parking_lot "This library provides implementations of Mutex, RwLock, Condvar and Once that are smaller, faster and more flexible than those in the Rust standard library" "Uncontended lock acquisition and release is done through fast inline paths which only require a single atomic operation. Microcontention (a contended lock with a short critical section) is efficiently handled by spinning a few times while trying to acquire a lock. The locks are adaptive and will suspend a thread after a few failed spin attempts." The only thing that I'm missing is how often the sleeping threads wake, as a ba

https://github.com/rust-lang/rust/blob/master/src/libstd/sys... Looks like it’s relying glibc for lock elision. Mind you the parking_lot::Mutex in the article is not the stdlib implementation, documented here: https://docs.rs/parking_lot/0.10.0/parking_lot/type.Mutex.ht... And that looks like it’s not using pthread, instead relying on primitives: https://github.com/Amanieu/parking_lot/blob/master/src/raw_m...