diamond-types

"rebase its own changes on top" is doing a lot of work and is essentially the core merge function of a CRDT. Also, the steps needed to get the rest of the way to a full CRDT is the solution to their logging woes: tracking every change and its causal history, which is exactly what is needed to exactly re-run any failing trace and debug it. Here's a modified version of the steps of their proposed solution: 1. For each document, every participating member holds the document, applied steps, and the current version. 2. A client submits (to the "server" or p2p) some transactional steps and the lastSeenVersion. 3. If the lastSeenVersion does not match the "server"/peer’s version, the client must fetch recent changes(lastSeenVersion). The server still accepts the changes. Both the client and the "server" rebase the changes of one on top of the other. Which one gets rebased on top of the other can be determined by change depth, author id, real-world timestamp, "server" timestamp, whatever. If it's by server timestamp, you get the exact behavior from the article's solution. If you store the casual history of each change, you can also replay the history of the document and how every client sees the document change, exactly as it happened. This is the perfect debugging tool! CRDTs can store this casual history very efficiently using run-length encoding: diamond-types has done really good work here, with an explanation of their internals here: https://github.com/josephg/diamond-types/blob/master/INTERNA... In conclusion, the article seems to be really down on CRDTs in general, whereas I would argue that they're really down on Yjs and have written 80+% of a CRDT without meaning to, and would be happier if they finished to 100%. You can still have the exact behavior they have now by using server timestamps when available and falling back to local timestamps that always sort after server timestamps when offline. A 100% casual-history CRDT would also give them much better debugging, since they could replay whatever view of history they want over and over. The only downside is extra storage, which I think diamond-types has shown can be very reasonable.

essed runs. If any part of the code works character-by-character, it'll become a bottleneck. Oh and did I mention that it works even better if you use columnar encoding, and break the data up into a bunch of small arrays? Yeahhhh. So thats why diamond types - my optimized egwalker implementation - is tens of thousands of lines of code instead of a few hundred. (Though in my defence, it also includes custom binary serialization, testing, wasm bindings, and so on.) Rust makes the implementation way easier to implement thanks to traits. I have simple traits for data that can be losslessly compressed into runs[1]. A whole bunch of code takes advantage of that, by providing tooling that can work with a wide variety of actual data. For example, I have a custom vec wrapper that automatically compresses items when you call push(). I have a "zip" iterator which glues together other iterators over run-length encoded data. And so on. Its great. Though now that I think about it, maybe all that trait foo is what makes it headache inducing. I swear its worth it. [1] Eg MergableSpan: https://github.com/josephg/diamond-types/blob/00f722d6ebdc9f...

locality. The trick is making one that understands ranges, not just singular keys like std BtreeMap. The code is here if you’re curious. It’s implemented on top of a pair of vecs - which makes it 100% safe rust, and curiously faster than the equivalent tree-of-raw-allocations approach. Please forgive the poor documentation - it’s an internal only type and the code is new enough that the paint hasn’t dried yet. https://github.com/josephg/diamond-types/blob/master/src/ost...

e. Libp2p, notably, has been rock solid for us and barely a time drain at all, apart from some unexpected interactions with Tor which are mostly about the lack of an official first-class Tor transport, which is specific to our use case and should start to change soon when Tor's Arti is ready. > and ultimately Nostr -- even though not truly P2P in that sense -- which already happens to have a first try going with nostrchat.io. Nostr and Bluesky both seem very promising for the open-world use case of social networking, and it has been amazing to see Nostr grow so rapidly as a community. I am rooting for this project and we might use it someday in Quiet for public feeds. Timed deletion is the user requirement that drives me away from building Quiet on Nostr. Based on conversations I've had with users doing sensitive work (and based on my own experience as a founder of Fight for the Future) timed deletion is extremely important to team security, and for deletion to be meaningful one needs more control over where the data is relayed than what Nostr provides in the default mode. A group that wanted trustworthy timed deletion would have to control their own private Nostr relay. Technically, a Tor relay could subvert the timed deletion of some Quiet messages just by capturing all traffic, but this is much less of a worry. > If P2P is something that is truly desired, I feel like projects like Briar (https://briarproject.org/how-it-works/) have solved this with Bramble (https://code.briarproject.org/briar/briar-spec/blob/master/p...) more eloquently than it could be done on top of IPFS. Bramble could work for us and I would recommend that anyone look into it. Briar is probably the most similar thing to Quiet that exists right now. There are big differences between Quiet and Briar, but we could definitely build Quiet on Bramble if it adequately supports iOS. My worry would be its maturity as a tool for people building things other than Briar. That could be worth the risk though and I do recommend anyone else reading this thread look at Bramble if you are doing something similar. > I could nevertheless imagine it being overtaken fairly quickly by other projects sporting a rather lightweight and more managable basis, that allows for increased development speed and ultimately for faster iteration on features that users might wish for (e.g. DMs, @-mentions, message deletion, mobile clients, you-name-it) -- without the need to invest heavily into e.g. performance (or reliability!) issues of the underlying framework. This is definitely something we will keep an eye on, and thank you for the thoughtful advice! My guess is that as soon as we have a significant number of real users we will need to build things that don't happen to be supported by whatever stack we choose (whether that is our current stack, Bramble, Veilid, Automerge, etc.) So the question is what's the easiest one to maintain and adapt. So far libp2p and IPFS have both been good in that department: implementations in many languages, active development, an absence of major problems showing signs of maturity (especially in libp2p), etc. Also, my 2 cents are (for anyone following along) that if I had to do this all over again I would use Tor + Libp2p + Automerge. Libp2p and Gossipsub are solid, flexible, and will be around a while. No need to reinvent the wheel. The conceptual framework behind Automerge and Briar/Bramble are pretty similar (sync state!) but the Automerge team exists to serve people building other apps, while the Bramble team mostly focuses on Briar AFAIK. What's nice about Automerge is that the community around it (Ink & Switch, Martin Kleppmann, and other academics) is all at the academic frontier, so the level of thought and anticipation of user needs that goes into their decisions is very thorough, even if the implementations lag behind the papers. If I was doing real-time p2p text editing I would also look at the Braid project (braid.org) and Seph Gentle's work on Diamond Types, since that's where the most thought has gone into the raw performance you need for text CRDTs that can handle large documents: https://github.com/josephg/diamond-types

e) Open source colaborative servers (coordination, persistance, presence): - https://github.com/ueberdosis/hocuspocus - https://github.com/partykit/partykit - https://github.com/firesync-org/firesync

ew(); But I think thats about it. Maybe there's more manually specified types in "normal" rust because most functions are smaller than that. But, it doesn't feel so bad. In this case I could probably even remove the explicit type annotation for that queue definition if I wanted to, but it makes the compiler's errors better leaving it in. [1] https://github.com/josephg/diamond-types/blob/66025b99dbe390...

b.com/josephg/diamond-types/blob/3eb48478fd879e... And here's a simple fuzz tester: https://github.com/josephg/jumprope-rs/blob/318e87d3aae1b2a0...