Improvements to the build development workflow

[kevin940726]

It's just a curated list that's been sitting in my head for too long and it's growing bigger and bigger. I just want to list them down here for visibility and to hopefully gather some feedbacks.

Note that these improvements should only affect development mode, so that the production builds stay intact. While some of these sound like breaking changes, we can add some feature flags or environment variables to require the developers to opt-in to them.

• Replace the 2-pass build steps in development workflow with single-pass (Experimental development-only single-pass webpack workflow #36736)
  The current development workflow (npm run dev) consists of roughly 2 parts: building the packages with Babel and bundling them with webpack. This 2-pass workflow isn't really efficient, we're essentially starting 2 watchers on our file system and writing double the code to disk. Instead, we can leverage webpack to do all these in a single pass in development mode so that the initial startup could be faster.
• Replace Babel with SWC
  This just seems like a low-hanging fruit we can do to just drop replace babel with SWC to decrease the build time. Note that we'd probably lose some features that the custom babel plugins provide, so we have to handle them gracefully.
• Prevent directly writing to disk
  IO read/write is quite expensive, we can instead writing them to memory to improve the performance. A bonus advantage is that there's no longer a stale build, since that whenever we change the files the build will be invalidated and recompiled.
• Enable hot reload/refresh
  React has official support of React Fast Refresh, while webpack can also offer CSS hot reload. This should dramatically improve our DX with a faster feedback cycle for simple changes like adjusting styles.

Thoughts? Prior arts we can take experience from? More ideas?
Hit the thumbs-up button and subscribe to this issue for more interesting ideas in the future 😜 .

[noisysocks]

Replace the 2-pass build steps in development workflow with single-pass

I didn't think this was possible due to how we have it set up where each package is loaded into WordPress as a seperate dependency e.g. @wordpress/components → build/components.js → wp-components. But happy to be wrong. npm run dev is indeed getting really slow as the project grows 🙂

[kevin940726]

I didn't think this was possible due to how we have it set up where each package is loaded into WordPress as a seperate dependency e.g. @wordpress/components → build/components.js → wp-components. But happy to be wrong. npm run dev is indeed getting really slow as the project grows 🙂

Hmm, not sure what you mean here. What we're currently doing is roughly this:

1. Build all packages into it's own build folder using Babel. So packages/a11y will build packages/a11y/build and packages/a11y/build-module etc.
2. Watch and rebuild everything in (1)
3. Start webpack watch and build all the packages into project's root build folder. However, the entries are packages/*/build-module (hence require (1))

Instead, I propose we skip (1) and (2) entirely, and just use webpack and babel-loader (or swc-loader) to build the packages from packages/*/src into project's root build folder. So every time we change something, we don't have to wait for Babel to output the build then wait for webpack to bundle them. We can just tell webpack to handle them all at once in memory. In theory, together with "Prevent directly writing to disk", this should significantly improve the performance especially for devices that don't have SSDs 😆 .

[gziolo]

To be able to use SWC or esbuild, we need to remove all custom Babel plugins first. The most important one would be to stop using our custom plugins for JSX transforms as proposed in #34221.

[gziolo]

There are a few ongoing explorations that are worth emphasizing in this discussion:

1. @youknowriad is exploring if there would be benefits of switching from npm to pnpm ("Fast, disk space efficient package manager" – alternative to npm):
   1. You can observe the work in Try pnpm #37324, where he already included first performance comparisons to npm 6.
   2. I have managed to update the branch from Update lock file format to version 2 (npm 8) #36041 that locally works with npm 8, so we could use it for comparison with pnpm.
2. @youknowriad is also looking at using SWC ("super-fast compiler written in rust" – alternative to Babel) in Build Tooling: Explore faster builds split into individual packages #37318. As part of the same effort, he wants to exercise how much we could align with the patterns promoted by tools for monorepo like NX or Turborepo that orchestrate build process for individual packages and optimize workflows through smart caching.
3. We now have experimental support for React Fast Refresh in @wordpress/scripts for the start command added in Scripts: Enable React Fast Refresh for block development #28273. It should also be possible to add the same capabilities to the Gutenberg build workflow, but it might require more thinking. We already have a new entry point for React Refresh coming from the webpack plugin and another entry point for React Refresh runtime provided by Facebook. Nest steps would be integrating the webpack dev server with the existing webpack config for the Gutenberg plugin. Another challenge is finding a viable way to include the Babel plugin for React Refresh in the workflow that transpiles WordPress packages.

Some of the topics from the list were discussed during the bi-weekly Core JS meeting on WordPress Slack (link requires registration at https://make.wordpress.org/chat):

https://wordpress.slack.com/archives/C5UNMSU4R/p1639494096224000

We agreed to continue explorations as follows:

• Once we have PRs for npm 8, pnpm, and whatever other tools folks want to try for managing packages, we plan to compare experience and performance before deciding which package manager will become the default one Gutenberg.
• We agreed to seek ways to remove custom Babel plugins (JSX Add jsx-runtime entry for the new automatic JSX transform #34221, JSON Use ESM JSON imports instead of adhoc json imports #34176, emotion) or find a way to make them work with SWC or another tool like esbuild ("An extremely fast JavaScript bundler" – written in Go). @luisherranz made a wish we could switch to the Rome Toolchain ("A linter, compiler, bundler, and more for JavaScript, TypeScript, HTML, Markdown, and CSS." – currently being rewritten in Rust) as soon as it production-ready.

[kevin940726]

Both (2) and (3) are mentioned in the original plan above, and they should be possible and easier to achieve if we have (1) (#36736) implemented. I can continue to work on them once we agree to go on this path.

While some of the famous build tools mentioned here are very tempting, let's move this step-by-step and not go all in to replace all of our build tools at the same time. That's why I have this plan to outline the possible experiments in each step. I hope we're on the same page here, so that we can direct our focus on the same thing and have a clear roadmap of what can be done in the future.

Might be worth noting that I've made all of the original plans above work in my local branches, so they should all be possible. I'm just trying to make the adoption path as smooth as possible.