seastar

ions in one system call, starting with epoll() some 20 years ago, and culminating with io_uring recently introduced to Linux. With these applications, it is pointless to speed up system calls or context switches, because these take a tiny percentage of the runtime. 2. They say a chain is as weak as its weakest link. This is even more true in many-core performance (due to Amdahl's law). Complex software uses many many OS features. If OSv speeds up context switches and system calls and networking (say) by 10%, but then some other thing the software does is 2 times slower than in Linux, it is very possible that OSv's overall performance will be lower than Linux. Unfortunately, this is exactly what we saw a few years ago when ScyllaDB (then "Cloudius Systems") was actively benchmarking and developing OSv: Many benchmarks we tried were initially slower in OSv than in Linux. When we profiled what happened, we discovered that although many things in OSv were better than Linux, one (or a few) specific things in OSv which were significantly efficient in OSv than in Linux. This could be some silly filesystem feature we never thought was very important but was very frequently used in this application, it could be that OSv's scheduler wasn't as clever as Linux's to handle this specific use case. It could be that some specific algorithm was lock-free in Linux but uses locks in OSv, so becomes increasingly worse on OSv with the more CPUs you have. The main point is that if an application uses 100 different OS features - Linux had hundreds of developers optimizing each of these 100 features for years. The handful OSv developers focused on specific features and made clever improvements to them - but the rest are probably less optimized than Linux. 3. Many-core development is hot When the OSv project started 7 years ago, it was already becoming clear that many-core machines were the future, but it wasn't as obvious as it is today. So OSv could get some early wins by developing some clever lock-reducing improvements to its networking stack and other places. But the Linux developers are not idiots, and spent the last 7 years improving Linux's scalability on many-core systems. And they went further than OSv ever got - they support NUMA configurations, multi-queue network cards, and a plethora of new ideas for improving scalability on many core systems. On modern many-core, multi-socket, multi-queue-network-card systems, there is a high chance that Linux will be faster than OSv. 4. Posix API is slow This is related to the first issue (of software having evolved on Linux), but this time for more "traditional" software and not really state-of-the-art applications using the latest fads like io_uring. This traditional software is using the Posix API - filesystem, networking, memory handling etc. that was designed decades ago, and make various problematic guarantees. Just as one example, the possibility to poll the same file descriptor from many threads requires a lock every time this file descriptor is used. This slows down both Linux and OSv, but not giving OSv any advantage over Linux, because both need to correctly support the same slow API. Or even the contrary - Linux and its hundreds of developers continue to come up with clever tricks for each of these details (e.g., use RCU instead of locks for file descriptors) while OSv's few (today, very few) developers only had time to optimize a few specific cases. So it's no longer clear that if raw performance is your goal, OSv is the right direction. OSv can still be valuable for other reasons - smaller self-contained images, smaller code base, faster boot, etc. For raw performance, our company (ScyllaDB) went on a different direction: The Seastar library (http://nadav.harel.org.il/seastar/, https://github.com/scylladb/seastar) allows writing high-performance applications on regular Linux, by avoiding or minimizing all the features that makes Linux slow (like context switches) or cause scalability problems on modern many-core machines. Initially Seastar ran on both Linux and OSv (with identical performance, because it avoided the heavy parts of both), but unfortunately today it is using too many new Linux features which don't work on OSv - so it no longer runs on OSv. [1] https://groups.google.com/g/osv-dev/c/0WveOhAsZEU/m/jqHyNs9l...

hared memory multithreading as a concurrency model to compete with anything, it's just too broken on a fundamental level. [1] https://github.com/scylladb/seastar/wiki/Memcached-Benchmark [2] https://dsf.berkeley.edu/jmh/papers/anna_ieee18.pdf