docs: add buck2 docs

Signed-off-by: Austin Seipp <[email protected]>

docs/buck2.md

@@ -0,0 +1,384 @@
+# Buck2 builds
+
+> [!NOTE]
+> This document is primarily of interest to developers. See also [Contributing]
+> for more information on how to contribute in general.
+
+There is experimental support for building `jj` with [Buck2], a hermetic and
+reproducible build system, an alternative to Cargo.
+
+> [!IMPORTANT]
+> Buck2 support is a work in progress, and is not yet complete. It is not
+> recommended for primary development use.
+
+## Why Buck2
+
+Jujutusu currently uses Cargo as its build system, much like the vast majority
+of Rust projects. Currently, Cargo is in no way a major strain on the project,
+but as we want to tackle more complex projects in the future some limitations
+are worth thinking about.
+
+### Multi-language support
+
+Most projects eventually evolve to include more than a single programming
+language and due to many constraints this is often fairly natural, but
+conceptually difficult to model.
+
+We have two cases where this is particularly relevant for different reasons, where
+we want to integrate other languages into our Rust-based project:
+
+#### Case 1: C dependencies
+
+For instance, today, we rely on several large C libraries which are effectively
+abstracted out by Rust libraries using complex `build.rs` scripts. This largely
+works out due to the (very high) quality of the Rust ecosystem and maintiners of
+these particular packages, but it does mean we are effectively at the mercy of
+an unbounded amount of C code that can be included in the blink of an eye.
+
+Furthermore, tracking and auditing C code carefully is a requirement for more
+advanced security and integration approaches like fuzzing. Being able to
+represent your C dependencies and manage them is crucial.
+
+#### Case 2: JavaScript UX integration
+
+But tomorrow, we would like to include HTTP components or Tauri projects in this
+repository, which would necessitate a way of incorporating JavaScript and more
+complex and refined build processes. This means we need to stitch together the
+build process of `npm` with `cargo` in a way that is easy for every developer to
+test. However, these tools have no way to relate their underlying dependency
+graphs in any way, and so the build process becomes overly crude approximations
+of `make` that are difficult to maintain.
+
+As a concrete example, the [diffedit3] program is a mixed Tauri/localhost-webapp
+that functions as a 3-way diff editor. We would like to include it in the
+Jujutusu repository, but it is difficult to do so in a way that reliably ensures
+we can build the components together. Our options are both suboptimal:
+
+- Commit prebuilt `diffedit3.js` output from a JS compiler
+  - Conceptually simple.
+  - But bloats the repository size, slowing development.
+  - Easy to get out of sync. You have to do it manually and add CI to check it.
+  - Impossible to audit; could be malicious! It's much better to represent the
+    build process as a *pure* pipeline.
+  - If you have to do this, **make it fully automated**.
+- Write wrapper tools to build the JavaScript as part of the `Cargo` process
+  - Easy to forget or misuse.
+  - Difficult to maintain an integration with `Cargo` anyway!
+  - Often doesn't generalize well to other languages or tools
+  - No matter what, a source of user complaints and occasionally painful
+    debugging sessions (out of date dependencies, missed inputs, etc.)
+
+[diffedit3]: https://github.com/ilyagr/diffedit3
+
+### Hermetic builds
+
+> [!INFO]
+> Buck2 builds of Jujutsu are not yet hermetically sound
+
+Buck2 is *hermetic*, meaning that the build graph (ideally!) encodes the
+relationships between all input files in the entire system. Hermetic builds are
+a powerful concept that can be used to ensure that the build process is fully
+reproducible and deterministic up to a very large size.
+
+Hermetic builds also allow for the process to be cached and served back to users
+as well, meaning that the time taken for a build can often be reduced to nearly
+zero even from a cold local cache.
+
+### Programmable design
+
+Buck2 is configured with Starlark, a deterministic and reproducible language
+that is a subset of Python (with types!). This means that the build process can
+be treated as a programming problem and designed in an open-ended way instead of
+designed around one concept about how a build should work.
+
+### Early movers advantage
+
+Most of the previous advantages are seen as only being most useful to those with
+lots of developers. The process is described as long and arduous and painful to
+port everything to a hermetic build.
+
+However, by waiting too late to make the transition to hermetic tools that
+represent the build as a pure graph, the pain of fixing this is made infinitely
+more acute by the process of time and accumulated debt.
+
+While most of our current daily problems are easily tractable and we are
+iterating easily with the tools we have, by exploring Buck2 early, we can avoid
+and eliminate obstacles up-front that would make the process far more difficult
+later. Even if we eventually choose some other build tool or none at all,
+representing the idea as a pure build graph can help guide overall design
+decisions.
+
+## Current support & feature parity
+
+Some notes about build compatibility are included below.
+
+### Overall status
+
+Legend:
+
+- ✅: Supported
+- ⚠️: Partial support/WIP
+- ❌: Not supported
+- ❓: Investigation needed
+
+| Feature                | Status |
+|------------------------|--------|
+| `rust-analyzer`        | ⚠️     |
+| CI setup (GHA)         | ❌     |
+| RBE/GHA Action Cache   | ❌     |
+| Hermetic toolchain     | ⚠️     |
+
+| Unique features        | Status |
+|------------------------|--------|
+| Auto `gen-protos`      | ✅     |
+| [OpenSSL-on-Win32](https://github.com/martinvonz/jj/pull/3554) | ❓ |
+
+### Known buck2 bugs
+
+NIH.
+
+### vs Cargo
+
+Legend:
+
+- ✅: Supported
+- ⚠️: Partial support/WIP
+- ❌: Not supported
+
+| Feature                | Cargo | Buck2 |
+|------------------------|-------|-------|
+| `rust-analyzer`        | ✅    | ⚠️    |
+| Fully working build    | ✅    | ✅    |
+| Debug/Release configs  | ✅    | ✅    |
+| Full test suite        | ✅    |️ ❌    |
+| Release-able binaries  | ✅    |️ ❌    |
+
+### Platform support
+
+Legend:
+
+- ✅: Supported
+- ⚠️: Partial support/WIP
+- ❌: Not supported
+
+| OS      | Architecture | Status |
+|---------|--------------|--------|
+| Linux   | x86_64       | ✅     |
+| Linux   | aarch64      | ✅     |
+| macOS   | x86_64       | ❌     |
+| macOS   | aarch64      | ⚠️     |
+| Windows | x86_64       | ✅     |
+| Windows | aarch64      | ❌     |
+
+## Step 1: Installing Dotslash
+
+We use [Dotslash] to manage Buck2 build tooling in a way that's consistent
+across all developers and amenable to version control. This helps ensure every
+developer gets consistent results (which is a big part of the selling point of
+hermetic build tools.)
+
+You can install Dotslash binaries by following the instructions at:
+
+- <https://dotslash-cli.com/docs/installation/>
+
+Or, if you have Rust installed, you can install Dotslash by running:
+
+```sh
+cargo install dotslash
+```
+
+Or, if you have Nix, you can install Dotslash with Nix by running:
+
+```sh
+nix profile install 'nixpkgs#dotslash'
+```
+
+## Step 2: Building `jj` with Buck2
+
+Assuming `dotslash` exists somewhere in your `$PATH`, you can now build `jj`
+with the included `buck2` dotslash file that exists under `./tools/bin`:
+
+```sh
+./tools/bin/buck2 run cli -- version
+```
+
+Dotslash will transparently run the correct version of `buck2`, and `buck2` will
+build everything needed to run `jj`.
+
+---
+
+## Buck2 crash course
+
+The following is an extremely minimal crash course in Buck2 concepts and how to
+use it.
+
+### Targets
+
+Buck2 is used to build **targets**, that exists in **packages**, which are part
+of a **cell**. The most explicit syntax for referring to a target is the
+following:
+
+```text
+cell//path/to/package:target-name
+```
+
+You normally use a so-called "fully qualified" name like the above as an
+argument to `buck2 build`.
+
+A cell is a short name that maps to a directory in the code repository. A
+package is a subdirectory underneath the cell that contains the build rules for
+the targets. A target is a buildable unit of code, like a binary or a library,
+named in the `BUILD` file inside that package.
+
+A fully-qualified reference to a target works anywhere in the source code tree,
+so you can build or test any component no matter what directory you're in.
+
+So, given a cell named `foobar//` located underneath `code/foobar`, and a
+package `bar/baz` in that cell, leads to a file
+
+```text
+code/foobar/bar/baz/BUILD
+```
+
+Which contains the targets that can be built.
+
+There are several shorthands for a target:
+
+- NIH.
+
+### Graphs: Target & Action
+
+NIH.
+
+### Package files
+
+NIH.
+
+### Mode files
+
+In order to support concepts like debug and release builds, we use the concept
+of "mode files" in Buck2. These are files that contain a list of command line
+options to apply to a build to achieve the desired effect.
+
+For example, to build in debug mode, you can simply include the contents of the
+file `mode//debug` (using cell syntax) onto the command line. This can
+conveniently be done with "at-file" syntax when invoking `buck2`:
+
+```sh
+buck2 build cli @mode//debug
+buck2 build cli @mode//release
+```
+
+Where `@path/to/file` is the at-file syntax for including the contents of a file
+on the command line. This syntax supports `cell//` references to Buck cells, as
+well.
+
+In short, `buck2 build @mode//file` will apply the contents of `file` to your
+invocation. We keep a convenient set of these files maintained under the
+`mode//` cell, located under [`./buck/mode`](../buck/mode).
+
+#### At-file syntax
+
+The `buck2` CLI supports a convenient modern feature called "at-file" syntax,
+where the invocation `buck2 @path/to/file` is effectively equivalent to the
+bash-ism `buck2 $(cat path/to/file)`, where each line of the file is a single
+command line entry, in a consistent and portable way that doesn't have any limit
+to the size of the underlying file.
+
+For example, assuming the file `foo/bar` contained the contents
+
+```text
+--foo=1
+--bar=false
+```
+
+Then `buck2 --test @foo/bar` and `buck2 --test --foo=1 --bar=false` are
+equivalent.
+
+### Buck Extension Language (BXL)
+
+NIH.
+
+## Examples
+
+Some examples are included below.
+
+### Run the Jujutsu CLI
+
+The following shorthand is equivalent to the full target `root//cli:cli`:
+
+```sh
+buck2 run //cli
+```
+
+This works anywhere in the source tree. It can be shortened to `buck2 run cli`
+if you're in the root of the repository.
+
+### Run BoringSSL `bssl speed` tests
+
+```sh
+buck2 run third-party//bssl @mode//release -- speed
+```
+
+### Build all third-party Rust dependencies
+
+```sh
+buck2 build third-party//rust
+```
+
+### Build all `http_archive` dependencies
+
+Useful for downloading all dependencies, then testing clean build times afterwords.
+
+```sh
+buck2 build $(buck2 uquery "kind('http_archive', deps('//...'))" | grep third-party//)
+```
+
+## Development notes
+
+Notes for `jj` developers using Buck2.
+
+### Build mode reference
+
+You can pass these to any `build` or `run` invocation.
+
+- `@mode//debug`
+- `@mode//release`
+
+### Cargo dependency management
+
+Although Buck2 downloads and runs `rustc` on its own to build crate
+dependencies, our `Cargo.toml` build files act as the source of truth for
+dependency information in both Cargo and Buck2.
+
+Updating the dependency graph for Cargo-based projects typically comes in one of
+two forms:
+
+- Updating a dependency version in the top-level workspace `Cargo.toml` file
+- Adding a newly required dependency to `[dependencies]` in the `Cargo.toml`
+  file for a crate
+
+After doing either of these actions, you can synchronize the Buck2 dependencies
+with the Cargo dependencies with the following command:
+
+```bash
+buck2 -v0 run third-party//rust:sync.py
+```
+
+This must be run from the root of the repository. Eyeball the output and make
+sure it looks fine before committing the changes.
+
+This step will re-synchronize all `third-party//rust` crates with the versions
+in the workspace Cargo file, and then also update the `BUILD` files in the
+source code with any newly added build dependencies that were added or removed
+(not just updated).
+
+### `rust-analyzer` support
+
+Coming soon.
+
+<!-- References -->
+
+[Contributing]: https://martinvonz.github.io/jj/latest/contributing/
+[Buck2]: https://buck2.build/
+[Dotslash]: https://dotslash-cli.com/