crdt-benchmarks

mentation. Some things can't be fast/low enough (e.g. time to load a document, time to apply updates, memory consumption, ..) other things only need to be adequate (e.g. time to insert a character into a document). Unfortunately, a lot of academic papers set a bad trend of only measuring one dimension. Yeah, it's really easy to succeed in one dimension (e.g. memory or insertion-time) and it is very nice click-bait. But that doesn't make your CRDT a viable option in practice. I maintain a set of benchmarks that tests multiple dimensions [1]. I'd love to receive a PR from you. [1]: https://github.com/dmonad/crdt-benchmarks

arks https://github.com/dmonad/crdt-benchmarks/issues/20

r as Immer). Hope you're still following along :) Maybe it helps to compare the TODO-MVC applications, as both SyncedStore (https://github.com/YousefED/SyncedStore/tree/main/examples) and @localfirst/state (https://github.com/local-first-web/state/tree/main/examples/...) have implemented these as examples! [1]: https://github.com/yjs/yjs [2]: https://github.com/dmonad/crdt-benchmarks [3]: https://github.com/immerjs/immer [4]: https://mobx.js.org/

t in any browser. The benchmarks confirm this. [2] Yjs is being used in practice by several companies (eg Nimbus Notes with >3 million users) for quite some time now. I'm not aware of any performance problems. [1]: https://blog.kevinjahns.de/are-crdts-suitable-for-shared-edi... [2]: https://github.com/dmonad/crdt-benchmarks

tead say something awful like: "Cool, but everything you did was done before. Even if they didn't make any of their work practical, usable or good they still published first and you obviously didn't do a good enough literature review if you didn't know that." And I feel an unfair defensiveness arise in me as a result that wants to find excuses to dismiss the work, even if the work might be otherwise interesting. Its hard to compare their benchmark results because they used synthetic randomized editing traces, which always have different performance profiles than real edits for this stuff. Their own university gathered some great real world data in an earlier study. It would have been much more instructive if that data set was used here. At a glance their RAM usage looks to be about 2 orders of magnitude worse than diamond-types or yjs. And their CPU usage... ?? I can't tell because they have no tables of results. Just some hard to read charts with log scales, so you can't even really eyeball the figures. So its really hard to tell if their work ends up performance-competitive without spending a couple days getting their enterprise style java code running with a better data set. Do you think thats worth doing? [1] https://github.com/dmonad/crdt-benchmarks [2] https://github.com/atom/teletype-crdt

ementation handles 2.5 million operations / second. A Rust/C implementation would surely handle more than that as it is not limited by automatic garbage collection. You can compare Automerge's current performance branch with Yjs' current implementation in [1]. To be fair, their implementation is not finished. If I understood correctly, they want to load the compressed format directly into memory. I played with this idea a few years ago and represented the Yjs model in an ArrayBuffer. This approach will improve load-time, but the performance will be intolerable in other aspects (e.g. when applying document updates, transforming the document to a String/JSON, or when computing diffs). The performance of Yjs is the result of more than five years research. I still have a couple of ideas to improve performance significantly. Although, my article clearly shows that it is definitely good enough now. [1]: https://github.com/dmonad/crdt-benchmarks/pull/4