Revision all-the-things with CRDT

[erlend-sh]

Let’s talk about our case for CRDT with Loro.

First of all, we want Revisions of posts (any page really) written in Weird, same as WordPress does. Except we wanna do it efficiently. The best way to do that, in the most Collectively Collaborative manner, seems to be CRDTs.

We believe this tech can give us a radically upgraded version of WordPress revisions, as well as a prose-centric version of pull requests later on.

---

https://loro.dev/docs/concepts/crdt

For ‘what is a CRDT’ from zick:

this is a pretty great slide-deck-type explanation of an eventually consistent replication mechanism very similar to what's used in Willow:

https://gun.eco/distributed/matters.html

I honestly don't totally understand the nuances of CRDTs at a technical / mathematic standpoint, but when I started to look into how they work I realized at a high level it was quite a bit simpler seeming than what I think it's often made out to be.

Different kinds of CRDTs are really just different algorithms for how you combine edits.

From Anselm (Jazz):

My best recommendation for where to start with building your own CRDT is http://archagon.net/blog/2018/03/24/data-laced-with-history/

the CRDTs in jazz.tools are a very straightforward implementation of that idea

Another go-to is Martin Kleppmann’s talks: https://www.youtube.com/playlist?list=PLeKd45zvjcDHJxge6VtYUAbYnvd_VNQCx&si=uw25GCI0uXENHHIR

---

None of the above needs to be understood in-depth however, thanks to general-purpose state management libraries like Loro - which is a fully librarized API already - we just need to figure out how to work with the Loro API in Weird.

We selected Loro because we seem to have very similar priorities/interests. For instance we really liked the sound of this:

Like Git, Loro saves a complete directed acyclic graph (DAG) of edit history. In Loro, the DAG is used to represent the dependencies between edits, similar to how Git represents commit history.

Loro supports primitives that allow users to switch between different versions, fork new branches, edit on new branches, and merge branches.

Based on this operation primitive, applications can build various Git-like capabilities:

• You can merge multiple versions without needing to manually resolve conflicts
• You can rebase/squash updates from the current branch to the target branch (WIP)

Shallow Snapshot is like Git's Shallow Clone, which can remove old historical operations that users don't need, greatly reducing document size and improving document import and export speed. This allows you to cold store document history that is too old and mainly use shallow doc for collaboration.

We believe that CRDTs can create a Git for Everyone and Everything:

It's for Everyone because by leveraging the power of CRDTs, we can make version control much easier to reason about and use for the average person.

It's (nearly) for Everything because Loro provides a rich set of data synchronization types. We're no longer limited to synchronizing plain text data, but can solve semantic automatic merging of JSON-like schema, which can meet most needs of creative tools and collaborative tools.

---

Use Case

Our first practical use case for Loro CRDT in Weird is our User Pages, like this one:

https://weird.one/muni.town/identity

We already support very basic revisions, but we want proper CRDT tracking.

[erlend-sh]

As a first step, a collaborator has successfully implemented history visualization: loro-dev/loro-prosemirror#24

480.mov