ci: build, test, lint only changed packages and dependants - #48
ci: build, test, lint only changed packages and dependants#48mikesposito wants to merge 2 commits into
Conversation
Currently, every CI run rebuilds all packages from scratch and runs tests, changelog validation, and ESLint across the entire monorepo, regardless of how many packages actually changed. This PR scopes TypeScript builds, tests, changelog validation, and ESLint to only the packages that changed plus their transitive dependants, by: - Using the GitHub Compare API to find the exact merge base between the PR head and the target branch. - Running `git diff --name-only` against that merge base to find changed files. - Expanding the changed set to transitive dependants (so type-correctness across package boundaries is preserved), and also to transitive dependencies when building (so `ts-bridge` has all referenced `dist/` outputs available). - Falling back to a full run when any file outside a package directory changes (e.g., root config files or workflow files), or when there is no merge base (push to `main`). A new `scripts/get-changed-workspaces.mts` script computes the changed package set and is called from the `prepare` job (24.x matrix run only). Downstream jobs read the outputs from `prepare` directly — the separate `get-changed-packages` job has been removed, saving one sequential job hop. The `changed-paths` output gates both ESLint scope and TypeScript build scope: when it is `"full"`, both run against all packages; when it is a JSON array of paths, each runs only against that subset. Four benchmark PRs target this branch to verify the expected behaviour: | PR | Change | Expected | |---|---|---| | [#9486](MetaMask/core#9486) | CI-only change (workflow file comment) | Full run — workflow files are not in `IGNORED_ROOT_FILES` | | [#9487](MetaMask/core#9487) | Single package (`@metamask/logging-controller`) | 3 packages (`logging-controller` + 2 transitive dependants) | | [#9488](MetaMask/core#9488) | Three packages (`accounts-`, `gas-fee-`, `network-controller`) | 38 packages (those 3 + transitive dependants) | | [#9489](MetaMask/core#9489) | `README.md` (ignored) + `logging-controller` | 3 packages — `README.md` does not trigger a full run | - [ ] I've updated the test suite for new or updated code as appropriate - [ ] I've updated documentation (JSDoc, Markdown, etc.) for new or updated code as appropriate - [ ] I've communicated my changes to consumers by [updating changelogs for packages I've changed](https://github.com/MetaMask/core/tree/main/docs/processes/updating-changelogs.md) - [ ] I've introduced [breaking changes](https://github.com/MetaMask/core/tree/main/docs/processes/breaking-changes.md) in this PR and have prepared draft pull requests for clients and consumer packages to resolve them <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Changes how every PR is validated; incorrect workspace expansion or root-change detection could skip builds or tests, though impact is limited to CI, not shipped product code. > > **Overview** > **CI runs only what the PR touches** instead of rebuilding and testing the whole monorepo on every run. > > The `prepare` job (Node 24.x) resolves the merge base via the GitHub Compare API, runs `get-changed-workspaces.mts`, and exports `package-names`, `changed-paths`, and `merge-base`. Downstream jobs use those outputs: changelog validation and per-Node test matrices iterate only affected workspace names; `wallet-cli` e2e runs only when that package is in the set. > > **ESLint** moves to a dedicated `lint-eslint` job that runs `yarn lint:eslint` on all packages when `changed-paths` is `full`, otherwise only on the listed workspace paths. > > **Build** uses `generate-partial-build-tsconfig.mts` plus `ts-bridge` for a partial TypeScript reference graph (changed packages plus transitive dependants and dependencies) when a merge base exists and there is no root-level change; otherwise it keeps `yarn build`. Root or workflow changes outside ignored files (e.g. not `README.md`) still force a full run; pushes without a merge base fall back to all packages. > > New shared logic lives in `scripts/lib/workspaces.mts` (git diff, dependency graph, root-change detection); `tsconfig.scripts.json` now includes `*.mts` for those scripts. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit b910c315f2667f29cca128d10fd37f2615d8b6c5. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
…ces in incremental CI (#9643)
Previously, any PR that touched `yarn.lock` would fall through to a full
rebuild and test run of all packages, because the script had no way to
know which packages were actually affected by the lockfile change.
Similarly, any change to the root `package.json` (including version-only
bumps from release PRs) triggered a full run.
This PR adds smart diffing for both files:
**`yarn.lock` diffing:** When `yarn.lock` appears in the changed files,
it is parsed using `@yarnpkg/parsers` and compared against the
merge-base version. Entry checksums are compared (rather than
resolutions) so that any change to installed package content is
detected, including the rare case of a package being re-released under
the same version. The changed package names are then cross-referenced
against each workspace's full transitive dependency closure (built by
walking the resolved lockfile graph via `@yarnpkg/core`) to produce the
minimal set of workspaces that need to be rebuilt and tested.
**Root `package.json` diffing:** When `package.json` changes, both sides
are parsed and compared with the `version` field stripped. A
version-only diff (e.g. a release PR bump) is ignored; any other change
(scripts, dependencies, etc.) still triggers a full run.
**Refactored `computeChangedWorkspaces`:** The function now fetches
workspaces and changed files internally, and returns `{ workspaces:
Workspace[], hasRootChange: boolean }` instead of `Set<string>`. This
removes boilerplate from both callers and allows
`get-changed-workspaces` to short-circuit before doing any workspace
computation when `hasRootChange` is true.
- Part of the incremental CI build work from #9373
- [ ] I've updated the test suite for new or updated code as appropriate
- [ ] I've updated documentation (JSDoc, Markdown, etc.) for new or
updated code as appropriate
- [ ] I've communicated my changes to consumers by [updating changelogs
for packages I've
changed](https://github.com/MetaMask/core/tree/main/docs/processes/updating-changelogs.md)
- [ ] I've introduced [breaking
changes](https://github.com/MetaMask/core/tree/main/docs/processes/breaking-changes.md)
in this PR and have prepared draft pull requests for clients and
consumer packages to resolve them
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes how CI decides what to build/test; incorrect lockfile or
package.json diff logic could skip needed workspaces or over-build. New
Yarn Berry APIs in scripts add maintenance surface but no runtime
product impact.
>
> **Overview**
> **Incremental CI** no longer treats every `yarn.lock` touch as a full
monorepo run. When the lockfile changes, the script diffs merge-base vs
head checksums, maps changed packages to workspaces whose **transitive**
closure includes them (via `@yarnpkg/core`), then applies the usual
dependant/dependency expansion.
>
> **Root `package.json`** is ignored for “root change” unless something
besides `version` changed (release-only bumps stay incremental).
`yarn.lock` is also excluded from the generic root-change path so
lockfile logic handles it.
>
> **`computeChangedWorkspaces`** now takes `{ mergeBase, headRef,
includeDependencies }`, loads workspaces and git diffs internally, and
returns `{ workspaces, hasRootChange }` with dependency graph edges as
`Workspace` objects. Callers `get-changed-workspaces.mts` and
`generate-partial-build-tsconfig.mts` were simplified to match.
>
> Adds `@yarnpkg/cli`, `@yarnpkg/core`, `@yarnpkg/fslib`, and
`@yarnpkg/parsers` dev deps (large `yarn.lock` churn). **`foundryup`**
gets a `@ts-expect-error` on `tar` extract `transform` for `tar@7.5.12`
typing.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
4bf933e12ddfbec50acecf0b6c175b9bd6bfce68. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
GitHub Actions
Inactive
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| CHANGED_PATHS="full" | ||
| fi | ||
| echo "package-names=$PACKAGES" >> "$GITHUB_OUTPUT" | ||
| echo "changed-paths=$CHANGED_PATHS" >> "$GITHUB_OUTPUT" |
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Caution MetaMask internal reviewing guidelines:
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
3 participants
Explanation
Cherry-picking these two commits from
coreto optimize CI:References
Checklist