Skip to content

infra: three submodules, watched releases, registry-mirrored skills - #358

Merged
marc0olo merged 14 commits into
mainfrom
infra/sources-strategy
Aug 27, 2026
Merged

infra: three submodules, watched releases, registry-mirrored skills#358
marc0olo merged 14 commits into
mainfrom
infra/sources-strategy

Conversation

@marc0olo

@marc0olo marc0olo commented Aug 26, 2026

Copy link
Copy Markdown
Member

Closes #355.

.sources/ held 16 submodules. Three are opened by the build; the rest were vendored so an agent could check a fact, or to supply skills through a symlink chain. Both have cheaper answers.

Result: 3 submodules, 10 watched entries across 6 upstreams, 7 verified on demand, skills mirrored from the registry.

Kept, because the build opens them

Submodule Resolved by
motoko synced pages, plus 52 <motokoExamples> includes
internetidentity sync-ii-spec.mjs, which generates two reference pages
examples remark-snippet.mjs, which extracts snippet= code

Thirteen go: eleven that existed only so an agent could check a fact, plus icskills and dotskills, which fed the skills symlink chain. The eleven appeared only in the AGENTS.md source table and .agents/submodule-bumping.md, and CI initializes just examples and motoko, so nothing in the build referenced any of them.

What is worth watching, and what is not

Watching everything meant a weekly check on repos the docs barely draw on. Measured footprint in docs/ decided each one:

Watched Pages Hits Also
icp-cli 51 461 the docs-site link slug
motoko-core 63 328
cdk-rs 47 336 pinned versions in prose
icp-js-core 15 47
icp-cli-recipes 11 26 every hit is a pinned recipe version in prose; one entry per recipe
icp-js-canisters 4 8 a stated minimum version

The other six became a reference group: icp-cli-templates (linked, not restated), candid (a rarely moving spec), response-verification (two pages, both linking to docs.rs), chain-fusion-signer (7 mentions), papi (1 page), ic-pub-key (3 pages). They keep their entry and verify path so an agent knows where to look, but carry no pin and produce no issue: verify against the latest release on demand, since the surface is small enough that the next edit to those pages catches drift. Each entry records the footprint that decided it, so promotion back to watched is a judgment someone can re-make.

Every upstream is now checked exactly once

Upstream Checked by Produces
motoko, internetidentity their existing sync workflows, unchanged a bump PR
examples upstream-releases.yml, vendored an issue when the gitlink falls behind master
10 watched entries upstream-releases.yml, watched an issue when a newer release exists
7 reference repos nothing nothing; verified against the latest release on demand
29 IC skills .claude/sync-ic-skills.sh on session start nothing; not committed

examples previously had no drift notification of any kind: it tracks a branch, and the only two workflows that could have noticed belong to the other two submodules. Issues carry upstream-sync plus a per-upstream label, and a run whose title already matches an open issue does nothing, so a repo that stays behind produces no weekly noise.

Watching

.sources/upstream.json records per repo: the ref the docs are verified against, where its releases appear, which file's diff is the review payload, and what to re-check. scripts/check-upstream-releases.mjs writes an issue body per repo that moved (no token needed). npm run check:upstream runs it locally.

Agents verify against the pinned ref, so a review is reproducible and a page cannot document a flag from a release whose links have not been adapted:

curl -sL https://raw.githubusercontent.com/dfinity/icp-cli/<pinned>/docs/reference/cli.md

raw.githubusercontent.com rather than gh api .../contents/...: the API returns base64 truncated at ~2.3 KB against 71 KB for the full file, and a truncated flag reference looks complete.

Where releases appear differs per repo, and getting it wrong fails silently

track Latest from Repos
release git tags matching an anchored pattern icp-cli, motoko-core, icp-js-core, and one entry per icp-cli-recipes recipe
crate crates.io cdk-rs
npm npm registry icp-js-canisters

Seven defects found and fixed: three while building this, one from Copilot, three from review:

  1. Lexicographic sorting reported downgrades. These repos tag in four shapes; as strings 0.9.4 beats 0.20.1. Sorting is numeric-run aware.
  2. cdk-rs was pinned to 0.20.1, which is not a tag but the published crate version, with the newest tag two minors behind. Against tags that pin could never be overtaken, so the repo governing every Rust code block would have reported "current" for ever. A release pin that is not among the matched tags is now a loud failure; every pin was audited.
  3. icp-js-sdk-docs was the wrong repo entirely. It is the js.icp.build delivery site, so a commit meant the docs site rebuilt. Replaced by icp-js-core (tags match npm) and icp-js-canisters (tags lag npm by two majors, so npm is the signal). Sibling repos, different correct answers.
  4. icp-cli-recipes monitoring was a no-op for four of five recipes. Refs compare by numeric run, so rust- against static-site- compares textually and only the alphabetically-last prefix could ever be reported, while the entry read as watched. Each recipe now has its own entry, pin, issue and label, which immediately surfaces four releases the docs are behind.
  5. The pin guard covered only the release track. A crate or npm pin in the wrong shape (v0.20.1 for a crate that publishes 0.20.1) compared as older for ever. Registry pins are now checked against the published version list, and crates.io reads max_stable_version so a prerelease is not reported as the latest release.
  6. summarizeChange counted by set membership, so an added line that already appeared elsewhere in the file read as unchanged. It now counts occurrences: cli.md v1.1.0 to v1.3.0 reports 14 added rather than 11.
  7. The commit track ordered short SHAs with the version comparator, so an update whose SHA happened to sort lower read as "current". Demonstrated with pin ef03698 against head 9f1e819. SHAs have no order, so it is now a prefix match, with the full SHA in links.

commit is currently used by nothing: it exists for a repo with no releases at all, and after the reduction below no watched repo is in that state.

First run: 9 of 11 have drifted

icp-cli v1.1.0→v1.3.0, motoko-core v2.4.0→v2.6.1, cdk-rs 0.20.1→0.20.2, icp-js-core v5.4.0→v6.1.0, icp-js-canisters 3.5.0→3.6.0, and four recipes: asset-canister v2.2.1→v2.3.0, rust v3.3.0→v3.4.0, motoko v5.0.0→v5.1.0, prebuilt v2.0.0→v2.1.0. The static-site recipe and examples are current.

Three carry real content risk: motoko-core (two minors of mo:core), icp-js-core (a major bump, Agent.readState revamped), cdk-rs. Where a changelog exists it is the review payload, because "check every API signature" is unbounded while a changelog names the symbols to grep for.

icp-cli is pinned at v1.1.0 deliberately: #354 moved the doc links to the 1.3 slug, but the commands were last verified against v1.1.0, so the first watch issue carries that pass rather than hiding it.

Skills

.claude/sync-ic-skills.sh on SessionStart, byte-identical to the script dfinity/examples runs, replaces two submodules and a double symlink chain. Coverage rises 16 → 29 and corrects two table entries pointing at skills that no longer exist upstream: asset-canisterstatic-site (which covers the legacy recipe and the migration) and vetkdvetkeys.

No symlinks, in either direction: Claude finds a skill whether it is a directory or a link, no other harness auto-discovers any path, and for the synced 29 a symlink is actively wrong because the sync consumes anything at a name the registry publishes (verified in a sandbox). AGENTS.md states the non-Claude path instead: fetch the registry index by URL, explicitly not .claude/skills/, which is gitignored and empty in a fresh clone.

icp-brand-design, icp-brand-voice, and technical-documentation stay committed. The last is not on the registry and changed once since our pin, only to relicense AGPL-3.0-only → MIT. That license is why it was a submodule; MIT still requires the notice to travel, so its text sits in the skill directory and a new root NOTICE records the attribution.

Instructions updated

AGENTS.md (source material, CLI verification, the 1.3 slug no longer derived from a submodule pin, skills, key directories), .agents/upstream-tracking.md replacing submodule-bumping.md, and CONTRIBUTING.md.

Worth challenging

  • The workflow itself has never run. The script is exercised locally; the Actions job (app token, label creation, supersede logic) is not.
  • examples costs 265 MB for one page today (6 snippets, 2 examples). Keeping it bets on <CodeExample> spreading.
  • ic-cdk-timers and ic-cdk-executor version separately from ic-cdk and are not tracked individually; the affects note tells a reviewer to check them.
  • setup.sh now prunes stale .sources/ directories. An existing clone keeps them after a submodule is removed, and a frozen .sources/icp-cli/ looks authoritative while being exactly what the verify-at-the-pinned-ref rule exists to prevent. Only untracked directories are touched.
  • The reference group trades a prompt for less noise. If nobody edits those pages, drift sits there unreported. That is the deliberate bet: six weekly checks on repos with a 1-to-3 page footprint cost more attention than they return.
  • The three committed skills are watched by nothing.
  • The non-Claude skill path is an instruction, not a mechanism. Claude gets skills regardless; every other harness only by compliance.

Found along the way, not fixed here

ic-cdk is pinned to four different versions across the docs: 0.19 in 8 places, plus 0.20, 0.17, and 0.17.1. The published crate is 0.20.2. That is a content bug rather than infrastructure, so it belongs with the cdk-rs watch issue rather than in this PR.

Follow-up

#279 is unblocked: of the 11 repos its "keep as-is" bucket names, only examples is still a submodule, and affects now carries the upstream-to-pages mapping.

After merge: trigger the workflow once so the drifts land as issues, batch the pin-only bumps, then handle icp-cli and motoko-core as their own PRs.

Merge order

These three are independent in intent but share one file, so they have a preferred order: #354, then #357, then #358.

PR Why here
1 #354 CLI docs links to the 1.3 slug Content only, no dependencies, and the smallest diff.
2 #357 clear the hand-written validate --all violations Touches scripts/validate.js, data-persistence.mdx, chain-key-tokens.mdx; no overlap with the other two. Merging it here drops validate --all on main from 11 errors to 7.
3 #358 .sources/ reduction Largest diff, and it shares docs/guides/canister-management/lifecycle.mdx with #354 (different lines: a CLI link there, a stale .sources/ pointer here). Last means at most one rebase, on a cleaner validate baseline.

The remaining 7 errors after #357 are all in docs/references/internet-identity-spec.md, which is generated by sync-ii-spec.mjs. They clear when dfinity/internet-identity#4284 merges and the next spec sync runs; that PR is independent of the order here.

Eleven repos were vendored as submodules purely so agents could check a fact
against them. None of their bytes reach the built site, so each one was a
recurring bump PR with nothing to show for it, and several drifted years of
releases behind unnoticed.

They are now watched instead: .sources/upstream.json records the ref the docs
are verified against, scripts/check-upstream-releases.mjs compares that against
the newest matching tag, and a weekly workflow opens one issue per repo that
moved, carrying the ref delta and a summary of what changed.

Agents verify against the pinned ref via raw.githubusercontent.com rather than
main, so a review stays reproducible and a page cannot document a flag from a
release whose doc links have not been adapted yet.
@marc0olo
marc0olo requested a review from a team as a code owner August 26, 2026 12:02
examples, icskills, and dotskills had no notification when their pin fell
behind: motoko and internetidentity are covered by their own sync workflows,
and the release check only looked at the non-vendored repos. They are now in
upstream.json as a vendored group whose pin is read from the gitlink, so git
stays the single source of truth for it.

icskills and dotskills are both currently behind.
…tion

Skills came from two submodules through a double symlink chain
(.claude/skills -> .agents/skills -> .sources/<submodule>/skills). They now
come from skills.internetcomputer.org via a SessionStart hook, matching
dfinity/examples: a differential sync keyed on each skill's published hash,
offline-safe, and it only prunes skills it installed itself.

That drops the icskills submodule, the symlink farm, and a watcher entry, and
raises coverage from 16 skills to 29 — including static-site, custom-domains,
encrypted-maps, service-discoverability, and mops-cli. It also corrects two
stale names in the AGENTS.md table: asset-canister no longer exists (static-site
covers it, including the legacy recipe and migration) and vetkd is now vetkeys.

technical-documentation is not on the IC registry. It changed once since the pin
and only to switch its license field to MIT, so its 11 text files are vendored
next to the two brand skills that were already committed, and the dotskills
submodule goes too.

Three submodules remain: motoko, internetidentity, examples.
@marc0olo marc0olo changed the title infra: reduce .sources/ to five submodules, watch the rest for releases infra: three submodules, watched releases, registry-mirrored skills Aug 26, 2026
MIT requires the copyright and permission notice to travel with a copy. The
skill's frontmatter recorded its license and source but not the notice itself,
so the upstream license text now sits alongside the files and NOTICE records
the attribution.
The session-start hook is Claude Code only and automatic skill loading is
harness-specific everywhere, so nothing loads skills automatically elsewhere.
What is portable is where to get them: the IC skills from the registry index by
URL, since they are gitignored and absent from a fresh clone, and the three
repo-maintained skills from their committed path.

Also records that the sync replaces a local skill whose name is later published
to the registry, so publishing one means deleting the local copy in the same
change.
…e version

The cdk-rs pin was 0.20.1, which is the published crate version, not a git tag:
the repo's newest bare-semver tag is 0.19.0, two minors behind. Compared
against tags, the pin could never be overtaken, so the check reported 'current'
forever for the repo that governs every Rust code block.

A release pin that is not among the matched tags is now a loud failure rather
than silence, and cdk-rs tracks crates.io, which is where its releases actually
appear. Auditing the other ten pins found no further cases.

motoko-core and response-verification now diff their changelogs, which names the
APIs that changed instead of leaving 'check every signature' as the whole task.
… label

Auditing every watched repo against its real publishing channel found a second
case beyond cdk-rs: icp-js-sdk-docs carries no tags, so its commit was standing
in for an @icp-sdk/core release. Tracking npm instead shows the docs are
verified against 5.4.0 while 6.1.0 is out, a major bump across every JS code
block. The pin was derived from public/core/versions.json at the previously
pinned commit rather than guessed.

Registry-tracked entries get a registry link instead of a GitHub compare view
and no file diff, because a package version is not a ref git can resolve; both
were producing broken links and a failed fetch.

Every issue now also carries upstream-sync, so they are findable as a class
rather than only per repo.
icp-js-sdk-docs is the delivery site for js.icp.build and carries the API docs
for six libraries as zips. A commit on it means the docs site was rebuilt, not
that an SDK released, so it was a proxy for the thing we actually document.

The two packages the docs use are watched directly, and they need different
signals: icp-js-core tags match its npm versions, so tags are authoritative and
its CHANGELOG is the review payload, while icp-js-canisters publishes 3.6.0 with
its newest semver tag at v1.0.0, so npm is the signal there.

Both pins come from what the docs assert: v5.4.0 from the mirror's
public/core/versions.json at the previously pinned commit, and 3.5.0 from the
stated minimum in references/application-canisters.md.
@marc0olo
marc0olo requested a lite review from Copilot August 26, 2026 15:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 72 out of 73 changed files in this pull request and generated 5 comments.

Comment thread scripts/check-upstream-releases.mjs Outdated
Comment thread .github/workflows/upstream-releases.yml Outdated
Comment thread .sources/upstream.json Outdated
Comment thread .sources/VERSIONS Outdated
Comment thread .agents/upstream-tracking.md
…ches

Addresses the Copilot review on #358.

The commit track ordered short SHAs with the version comparator, so a real
update whose SHA happened to sort lower was reported as current. Demonstrated:
pinned ef03698 with head 9f1e819 returned 'current'. SHAs have no order, so the
pin either prefixes the head or it does not, and links now use the full SHA
while only the display is shortened.

The link row was inferred from whether a registry URL was set, which mislabelled
the commit track's GitHub compare view as 'Registry'; it is now named explicitly,
and the verify-diff guard keys off that name rather than the URL.

Also: upstream.json documented only two of the four tracks, VERSIONS claimed
7-char hashes while internetidentity uses 8, the label wording omitted both the
shared upstream-sync label and the vendored form, and the workflow's jq filter
uses an explicit index.
Twelve watched entries meant a weekly check on repos the docs barely draw on.
Measured footprint in docs/ decided each one:

  icp-cli           51 pages, 461 hits, plus the link slug
  motoko-core       63 pages, 328 hits
  cdk-rs            47 pages, 336 hits, plus pinned versions in prose
  icp-js-core       15 pages,  47 hits
  icp-cli-recipes   11 pages,  26 hits, all pinned recipe versions in prose
  icp-js-canisters   4 pages,   8 hits, with a stated minimum version

The other six move to a  group: templates (linked, not restated),
candid (a rarely-moving spec), response-verification (two pages that link to
docs.rs), chain-fusion-signer (seven mentions), papi (one page), ic-pub-key
(three pages). They keep their entry and their verify path so an agent knows
where to look, but carry no pin and produce no issue: verify against the latest
release on demand, since the surface is small enough that the next edit to those
pages catches drift.

First run drops from ten issues to five.
@marc0olo

Copy link
Copy Markdown
Member Author

All five Copilot comments addressed. One was a real bug in the script; four were accurate documentation mismatches.

1. commit track ordered SHAs (real bug). Confirmed by construction: with the pin at ef03698 and head 9f1e819, the version comparator returned "current", so a genuine update would have gone unreported. Commit SHAs have no order, so it is now a prefix match against the pin, and links use the full SHA while only the display is shortened. Fixing that exposed a second defect of my own: the link row was inferred from whether a registry URL was set, so the commit track's GitHub compare view was being labelled "Registry". The row is now named explicitly and the verify-diff guard keys off that name.

2. jq portability. Changed to an explicit index. Worth noting the concern was speculative: --jq 'first // empty' runs in production in dfinity/icskills' own sync workflow, and gh --jq uses an embedded engine rather than the runner's jq. The change costs nothing, so it is in.

3. upstream.json documented only two of the four tracks. Correct: crate and npm were added after that comment was written, and vendored entries use branch. All four are now documented.

4. VERSIONS claimed 7-char hashes. Correct: internetidentity uses 8. Relaxed to "short commit hash", since the length carries no meaning.

5. Label wording. Correct and then some: it omitted both the shared upstream-sync label and the upstream-submodule-<name> form used for vendored entries. All three are now described.

Unrelated to the review, the watch list also shrank from 12 repos to 6 after measuring what each upstream actually underwrites in docs/; the other six became a reference group with no pin and no issue. See the "What is worth watching, and what is not" section.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 72 out of 73 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

AGENTS.md:220

  • This section says there are "five" vendored submodules, but the table immediately below lists three (motoko, internetidentity, examples). This will mislead anyone following the instructions.
    scripts/check-upstream-releases.mjs:148
  • summarizeChange uses Sets of lines, which can undercount changes when the same non-empty line appears multiple times (duplicates in changelogs/headings are common). Using per-line counts will make the added/removed totals and heading detection more accurate.

Comment thread scripts/check-upstream-releases.mjs
Comment thread scripts/check-upstream-releases.mjs
Addresses the second Copilot review on #358.

The vendored check had both full SHAs in hand but put 7-char prefixes in the
compare link and, more importantly, in the git checkout command it asks a
maintainer to run. Short SHAs are now display-only.

The commit track's comment also claimed links resolve against the full SHA,
while the pin side of the compare URL is whatever upstream.json records. The
comment now describes that: the head side is full, the pin side stays short for
readability, and GitHub resolves a prefix on either side.
@marc0olo

Copy link
Copy Markdown
Member Author

Both comments from the second review are accurate and fixed in ebce87f.

Short SHAs in the vendored check. Correct, and the git checkout line was the part that mattered: the function already had both full SHAs in hand and was handing a maintainer a 7-char prefix to run. Full SHAs now go into the compare link and the checkout command, and short ones are display-only in the table and the issue title. Verified by forcing a mismatch, which produces checkout 217f8bdca0b29a56ba2355b2441a3123532c1b10 rather than checkout 217f8bd.

The comment that no longer matched the code. Also correct, and it was mine from fixing the previous review: I wrote "links resolve against the full SHA" while the pin side of the compare URL is whatever upstream.json records, which is short by design so the file stays readable. Rather than require full SHAs in config, the comment now describes the actual behaviour: the head side is full, the pin side is a prefix, and GitHub's compare view resolves a prefix on either side (confirmed against a live URL).

1. AGENTS.md said "five repos" over a table of three.
2. setup-node was unpinned while every other action in the repo is SHA-pinned.
3. The dispatch input was interpolated into a run: body; it now goes through env
   and is quoted.
4. icp-cli-recipes monitoring was a no-op for four of five recipes: refs compare
   by numeric run, so `rust-` against `static-site-` compares textually and only
   the alphabetically-last prefix could ever surface. Each recipe now has its own
   entry, pin, issue and label, which immediately surfaces asset-canister v2.2.1
   to v2.3.0, rust v3.3.0 to v3.4.0, motoko v5.0.0 to v5.1.0 and prebuilt v2.0.0
   to v2.1.0, all previously invisible.
5. scripts/setup.sh prunes .sources/ directories left behind by submodules the
   repo no longer vendors. A stale .sources/icp-cli/ is exactly the trap the
   verify-at-the-pinned-ref rule exists to close.
6. The pin guard covered only the release track. A crate or npm pin in the wrong
   shape compared as older for ever, so registry pins are now checked against
   the published version list, and crates.io reads max_stable_version so a
   prerelease is not reported as the latest release.
7. summarizeChange counted by set membership, so an added line that already
   appeared elsewhere read as unchanged. It now counts occurrences: cli.md
   v1.1.0 to v1.3.0 reports 14 added rather than 11.
8. Stale .sources/ paths in three pages, and migrating-from-dfx.md, which does
   not exist, named in AGENTS.md and CONTRIBUTING.md.
9. vincentkoc/dotskills is recorded as a reference entry so the vendored skill's
   origin is not lost.
The validator skips HTML comments but not MDX ones, and the fix for the stale
.sources/ pointer used an em-dash, which the repo bans.
@marc0olo

marc0olo commented Aug 27, 2026

Copy link
Copy Markdown
Member Author

All ten items addressed in 15f8c55 and 142ee5b. Thank you for the independent verification of the byte-identical skill, the MIT provenance, the registry contents, and the CI paths; that saved re-proving them.

1 to 3, the pre-merge fixes. AGENTS.md now says three repos over a table of three. setup-node is pinned to 49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0, the same SHA build.yml uses. The dispatch input goes through env: REPO: and is quoted in the script rather than interpolated into the body.

4, the recipes no-op. Correct and the most valuable catch here. Split into one entry per recipe, each with its own pin, issue, label and title. That immediately surfaces four releases the docs are behind, all previously invisible: asset-canister v2.2.1→v2.3.0, rust v3.3.0→v3.4.0, motoko v5.0.0→v5.1.0, prebuilt v2.0.0→v2.1.0.

One correction to the diagnosis, with evidence: asset-canister is not a prefix that never existed. It has five tags, a current release asset-canister-v2.3.0 published 2026-07-30, and a recipes/asset-canister/ directory on main. assets is the legacy prefix, with one tag (assets-v1.0.0) alongside the moving *-latest tags. The five recipe directories on main are asset-canister, motoko, prebuilt, rust, static-site, and those are the five entries. The rest of your point stands exactly: textual prefix comparison meant only static-site-* could ever be reported.

5, post-merge leftovers. scripts/setup.sh now prunes any .sources/ directory not declared in .gitmodules, warns per removal, and points at the pinned-ref rule. Tested against fake stale icp-cli/ and candid/ directories. Only untracked directories are touched. Worth noting my own clone had already dropped them, since git rm cleaned the working tree; the case that needed fixing is a third party pulling the merge, which is what you described.

6, the guard covered only tags. Fixed both halves. Registry pins are now validated against the published version list, so v0.20.1 for a crate that publishes 0.20.1 fails loudly instead of comparing as older for ever. crates.io reads max_stable_version first, so a prerelease is not reported as a release.

7, summarizeChange undercounted. Fixed to count occurrences rather than set membership. cli.md v1.1.0→v1.3.0 now reports 14 added, matching the real diff, where it previously said 11.

8, stale pointers. Fixed in lifecycle.mdx, candid.mdx, and governance/testing.md, and migrating-from-dfx.md is gone from AGENTS.md and CONTRIBUTING.md. That fix then introduced an em-dash in an MDX comment, which the validator checks (it skips <!-- --> but not {/* */}), so there is a follow-up commit for that. validate --all is back to the 11 pre-existing errors, none from this branch.

9, the judgment call. Agreed, and it is the part I would want a second opinion on too. Two things now recorded rather than left implicit: vincentkoc/dotskills is a reference entry, so the vendored skill's origin and the re-copy procedure survive, and the "watched by nothing" gap stays in the challenge list. The deeper point, that an upstream skill rewrite reaches agents with zero review, is inherent to the registry model; the alternative is pinning skills, which is what this PR removes. Worth revisiting if a skill ever regresses.

10, smoke test. Cannot run yet: workflow_dispatch requires the workflow file on the default branch, so the first run has to be post-merge. Plan is to dispatch it for one repo immediately after merging, before the Monday cron, since the app token and label creation have never executed. Noted at the top of the challenge list.

The prune tested for 'not declared in .gitmodules', which today is the same set
but would delete a scratch directory parked under .sources/ without warning. It
now also requires the directory to be empty or hold a .git entry, which every
submodule working tree does; anything else is reported and left in place.
@marc0olo

Copy link
Copy Markdown
Member Author

Thanks for re-verifying against the commit rather than the reply, and for correcting the tag-list truncation openly. The sort | tail -25 cut is a good cautionary detail; it is the same shape as the bug it sat next to, where a truncated view of an ordered list produced a confident wrong conclusion.

The prune comment. I took it further than wording, because the hazard you named is real even if today's .sources/ cannot hit it. The prune now requires two conditions: undeclared in .gitmodules and recognisable as a former submodule working tree, meaning empty or holding a .git entry. Anything else is reported and left in place. Tested with three directories in one run:

! removed stale .sources/empty-leftover (no longer a submodule)
! removed stale .sources/icp-cli (no longer a submodule)
! left .sources/my-scratch alone: not declared in .gitmodules and not a submodule working tree

2fb9919.

Merge order. Added as a section to all three PR bodies, so whoever merges does not have to reconstruct it. One detail worth recording there: #354 and #358 both touch docs/guides/canister-management/lifecycle.mdx, on different lines (a CLI link in #354, the stale .sources/cdk-rs pointer here), which is the only file any two of the three share. #357 overlaps with neither. So #358 last means at most one rebase, on a validate baseline already reduced from 11 errors to 7 by #357.

@marc0olo
marc0olo merged commit 4174594 into main Aug 27, 2026
8 checks passed
@marc0olo
marc0olo deleted the infra/sources-strategy branch August 27, 2026 12:40
marc0olo added a commit that referenced this pull request Aug 27, 2026
…aced (#370)

Closes #360, #361, #362, #363, #364, #365, #366, #367, #368, #369.

The first batch from the upstream watcher added in #358, plus one fix to
the watcher itself that running the batch exposed. **23 files, 18 of
them docs pages.** Two of the ten issues needed real content changes;
the other eight were version strings or pin bumps.

Every claim below was checked against the upstream at the new ref. Where
a replacement is named, it is upstream's own wording, not an inference.

## What a reviewer should scrutinise

1. **The 20 Motoko call-site replacements** are the only changes that
alter code a reader copies. Each one is named in upstream's
`@deprecated` marker, quoted below.
2. **`icp canister status <name> -i`** replacing `icp canister id
<name>`. This is a real command substitution, verified against the
installed CLI, not a rename.
3. **The examples path filter** decides whether a whole class of issue
fires. Its two-way verification is in the table below.

Everything else is a version string or a pin.

## motoko-core v2.4.0 → v2.6.1: 20 deprecated call sites (#362)

v2.6.0 deprecated every `Module.fromX` conversion that has a `toX`
counterpart. Our examples used five of them, so left alone each one
emits a compiler warning for anyone who copies the code. The
replacements are read from `src/*.mo` at v2.6.1:

| Deprecated | Upstream's `@deprecated` text | Sites |
|---|---|---|
| `Nat64.fromNat` | Use `Nat.toNat64` instead. | 7 |
| `Nat8.fromNat` | Use `Nat.toNat8` instead. | 5 |
| `Blob.fromArray` | Use `Array.toBlob` instead. | 5 |
| `Float.fromInt` | Use `Int.toFloat` instead. | 2 |
| `Text.fromChar` | Use `Char.toText` instead. | 1 |

**Import policy: four added, zero removed.** A replacement can need a
module the block did not import (`Nat`, `Char`), so those are added.
Nothing is removed, deliberately: an unused import is harmless, a
missing one does not compile, and pre-existing imports are not this
change's business. A first attempt did prune them and got it wrong three
ways, dropping `Text` where `hex.chars()` still needs it in scope for
contextual dot, plus `Cycles` and `Principal`, which are unrelated. That
pass was reverted.

Two cases handled by hand: one `Nat64.fromNat` appears in prose rather
than a fence (`ledgers.mdx`), and one fence is a fragment with no import
block of its own, so it gets the call change without an import.

## icp-cli v1.1.0 → v1.3.0: nothing lost, two commands that never
existed (#361)

The decisive check is that **`docs/reference/cli.md` lost 0 lines**
between the two refs (14 added, all for the new `icp canister link`). No
documented command or flag disappeared, so the bump itself is safe.

Verifying every `icp` invocation in `docs/` against that reference then
surfaced two commands that do not exist. Both confirmed against the
installed v1.3.0, not just the reference:

| In the docs | Actual | Sites |
|---|---|---|
| `icp canister id <name>` | `icp canister status <name> -i` (`-i` is
documented as "Only print the canister ids") | 4 |
| `icp identity whoami` | `icp identity principal` | 1 |

These predate the bump. They are exactly what the verification pass
deferred in #354 was for.

## The remaining eight

- **Four recipes** (#366, #367, #368, #369) each shipped **one identical
additive change**, an optional `visibility` field on metadata entries,
so the `icp.yaml` examples stay valid and only the 20 version strings
move: `asset-canister` v2.2.1→v2.3.0 (10 places), `rust` v3.3.0→v3.4.0
(6), `motoko` v5.0.0→v5.1.0 (3), `prebuilt` v2.0.0→v2.1.0 (1).
- **icp-js-core v5.4.0 → v6.1.0** (#364) is a major bump, and still
pin-only for us. It changes `Agent.readState`, delegation permissions,
effective subnet ID and URL path handling. Our examples import `agent`,
`agent/canister-env`, `principal`, `identity` and `candid` only, never
construct a `Delegation`, and never call `readState`.
- **cdk-rs 0.20.2** (#363) only makes entry points lenient about an
empty input vector. Pin-only.
- **icp-js-canisters 3.6.0** (#365) adds NNS converter fields and
snapshot visibility; the stated `>= 3.5.0` minimum still holds.
Pin-only.
- **examples** (#360) bumped to `42c474d`, with all six `snippet=`
regions verified to resolve.

## Watcher fix: report an examples bump only when it touches a quoted
file

Running the batch showed #360 was noise, and the first attempt at fixing
that was wrong in an instructive way. I dropped the check entirely,
reasoning that the build already fails when a `snippet=` path or
`#region` marker stops resolving. It does, but that only covers
**breakage**. It cannot see an example being **corrected upstream while
still resolving**, which leaves the docs quoting old code with a green
build. That is the more valuable signal, and dropping the check would
have lost it.

So the check stays and gets precise. The entry carries `pathFilter:
"snippets"`, and the check intersects the files changed between the
pinned gitlink and the branch head with every file a `snippet=` quotes,
opening an issue only when they overlap, naming the files.

Verified both ways against real history:

| Pin | Range | Result |
|---|---|---|
| `01edb5d8`, before a fix to the two `send_http_*` examples | 5
commits, 80 files | **reports**, naming
`motoko/send_http_get/backend/main.mo` and
`motoko/send_http_post/backend/main.mo` |
| `b4fe175`, the pin this PR bumps from | 4 commits, 78 files, none
quoted | **silent** |

The second row is the point: #360 fired for a range touching nothing the
docs quote, and would not fire now.

Two implementation details worth a look. The snippet path is
`<lang-dir>/<attribute>`, where the directory comes from the **fence
language**, so `LANG_TO_DIR` is mirrored from
`plugins/remark-snippet.mjs` rather than imported, because that plugin
pulls in remark dependencies and the workflow installs none. The two
must be kept in step, and an unmapped language throws rather than
silently matching fewer files. Separately, when the comparison cannot be
trusted, the API failing or truncating at its 300-file cap, the check
reports instead of assuming nothing moved.

The per-recipe split from #358 is unchanged. A batch release produces
one issue per recipe, four here for a single upstream change. That is
the cost of tracking them independently, and it is what made four stale
version strings visible.

## Verification

```
npm run check:upstream   # every entry current, exits 0
npm run build            # passes
npm run validate         # 7 errors, unchanged from main
```

The 7 are all in `docs/references/internet-identity-spec.md`, which is
generated. They clear when Internet Identity cuts its next dated release
containing
[dfinity/internet-identity#4284](dfinity/internet-identity#4284);
our pin is already on the newest existing release tag, so there is
nothing to sync until then. Tracked in #356.

## Known limits

- The two CLI command fixes are verified against `icp 1.3.0` locally and
against the published reference, but not executed end to end against a
live project.
- `snippet=` coverage depends on `LANG_TO_DIR` staying in step with the
plugin. The two now match key for key, verified by parsing both maps
rather than reading them, and an unmapped language fails loudly, which
is the safe direction. It is still a duplicated constant: the plugin
cannot be imported here because it pulls in remark dependencies the
workflow does not install.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

infra: reduce .sources/ submodules to what the site actually needs, watch releases for the rest

3 participants