diff --git a/llp/0049-hypignore-usage-policy.spec.md b/llp/0049-hypignore-usage-policy.spec.md index 405e25b3..921db6e0 100644 --- a/llp/0049-hypignore-usage-policy.spec.md +++ b/llp/0049-hypignore-usage-policy.spec.md @@ -68,6 +68,16 @@ matched over the set of spellings that denote it (as-given plus symlink-resolved), and the most restrictive verdict any spelling produces wins, so a `.hypignore` cannot be escaped by reaching its subtree through a symlink. +**Extended-by: +[LLP 0050 §normalization](./0050-ignore-enforced-in-adapters.decision.md#normalization)**: +symlinks are not the only way a filesystem spells one directory several ways. +The ancestor walk is unaffected by the rest (it `stat`s each candidate rather +than comparing two strings), but the machine-local list's membership test +compares a `cwd` a client reported against a `dir` a CLI declared, so each of +the spellings above is compared in a **folded** form (Unicode-normalized, and +case-folded on a volume probed case-insensitive). Like canonicalization, the +fold can only ever make the verdict more restrictive. + ## Classes {#classes} | Class | V1 | Meaning | diff --git a/llp/0050-ignore-enforced-in-adapters.decision.md b/llp/0050-ignore-enforced-in-adapters.decision.md index 22d145f9..5250da5e 100644 --- a/llp/0050-ignore-enforced-in-adapters.decision.md +++ b/llp/0050-ignore-enforced-in-adapters.decision.md @@ -194,6 +194,13 @@ governs this directory?" is `scopeGoverns`, which has to be the same predicate `resolve` used, or `policy show` names a governor the gate did not use and `policy unset` refuses to remove an entry the gate is enforcing. +Symlinks are not the only way a filesystem spells one directory several ways. +[§normalization](#normalization) widens the same set again, by Unicode +normalization and per-volume case, through the same two-pass guard: the two are +one mechanism in the code, not two stacked ones. Read that section before +touching `selectGoverning`, and note that the fold it adds stops at the gate +while the canonicalization described here does not. + ## Why not the gateway - The gateway is the **provider-agnostic** proxy ([LLP 0016](./0016-ai-gateway.decision.md)). @@ -212,6 +219,222 @@ Two copies of a privacy-critical matcher drift apart. A single core module with one test suite is the safer home; sibling-to-sibling plugin imports would be worse coupling than both importing core. +## The set of spellings that denote a directory is volume-dependent {#normalization} + +The shared matcher compares **strings**. A filesystem hands one directory +several strings, and which mechanisms apply is a property of the **volume**, +not of the path and not of the platform: + +| mechanism | folded by | volume-dependent? | +|---|---|---| +| symlinked components | `realpath(2)` | no | +| Unicode normalization (NFC vs NFD) | nothing in `node:fs` | yes, and folding is **unsafe** where it does not apply, but harmless *at the gate* (see below) | +| case | nothing in `node:fs` | yes, and folding is **unsafe** where it does not apply | + +`realpath(2)` resolves symlinks and does nothing else. On a default macOS +(APFS) volume the kernel accepts, `stat`s, and `chdir`s to spellings it will not +fold: `Proj` and `proj` are one directory, and `Café` spelled NFC (U+00E9) and +NFD (`e` + U+0301) are one directory. So the gate could be handed a `cwd` whose +spelling differs from the spelling a machine-local entry was declared with and +return `full` for a directory the user opted out of. That is not an exotic +case: macOS frameworks and Finder-derived paths emit NFD while typed and +JSON-transported paths are usually NFC, and the two paths this gate compares are +produced by **different processes at different times** (a CLI resolving a mark, +versus a client reporting a `cwd`). + +**Decision: list membership compares a folded spelling of both sides.** The fold +is `src/core/usage-policy/fold.js`: + +1. **NFC unconditionally, and only because this is the gate.** It is a total + function of the string, needs no filesystem access, cannot fail, and is the + identity on a path that is already composed. What makes it safe here is + **not** that NFC and NFD always name one directory. They do not: on every + Linux volume this codebase targets, `caf` + U+00E9 and `cafe` + U+0301 are + two directories with two inodes, and both can exist in one parent + (demonstrated on an ext4-backed overlay host: distinct `ino`, distinct + contents, both listed by `readdir`). Folding them together therefore *can* + merge two genuinely different directories, exactly as unconditional case + folding would. + + It is safe at the gate anyway, for a reason specific to the gate: the + resolved class is `max(declared, folded)` (`selectGoverning`, and the argmax + discussion below), so a fold that merges two distinct directories can only + ever **over-restrict**, i.e. decline to record a directory that was in fact + permitted. That is a usability cost + and never a privacy or data-loss one. Case is put behind a probe rather than + given the same treatment because case aliasing is far more likely to collide + with a real, deliberately-distinct sibling (`Makefile` vs `makefile`) than a + normalization difference is, not because NFC folding is universally sound. + + **Do not reuse `foldPath` in a predicate where widening is not free.** In a + *deletion* predicate (`hyp purge`) or a *disclosure* predicate, widening + removes or reveals rows for a directory the user did not name, and the + `max()` argument above does not apply. Closing the purge seam below needs + either a per-volume normalization-insensitivity probe (no such probe exists; + the current one answers only the case question) or a darwin-only guard. + See "Not covered". +2. **Case only behind a per-volume probe.** Case-sensitivity is a property of + the mounted volume: an APFS volume can be formatted case-sensitive and every + ext4 volume is. Folding it unconditionally would merge two genuinely + different directories, which is a correctness bug in the other direction. The + probe compares the `dev`/`ino` of a path against a case-flipped spelling of + its last segment, memoizes by `dev`, and is inert (constant `false`, no + syscall) off darwin. An undetermined probe resolves to "case-sensitive", + which is the pre-fold behaviour, so a failed probe can only fail to *add* + reach. + +The fold must **distribute over the path separator**, since its only consumer is +a segment-aware prefix test: `fold(a + '/' + b) === fold(a) + '/' + fold(b)`. +Both halves do (`/` is a starter that participates in no canonical composition, +and `toLowerCase` maps it to itself), and the property is asserted rather than +assumed. + +### A widened spelling must only ever add restriction + +Producing a folded spelling is necessary but **not sufficient**. The +machine-local list's nearest-governs step is an **argmax over match depth**, and +an argmax discards verdicts instead of merging them. A less restrictive entry +that gains reach through its folded spelling can become the deepest match and +displace a broader restrictive entry that already governed: a `--sync` carve-out +spelled NFC would punch a hole in a private tree spelled NFD, and the directory +would **start recording and forwarding**. Nothing about "compare folded +spellings" prevents that on its own. + +So nearest-governs is evaluated **twice**, once over the spellings exactly as +declared (which reproduces the pre-fold verdict) and once folded, and the **more +restrictive of the two answers wins**, the declared one breaking a class tie +because it is the spelling the user typed. The resolved class is therefore +`max(pre_fold, folded)` on the restrictiveness lattice by construction, which +makes "folding never opens the gate" structural rather than a property someone +has to remember +([LLP 0049 §fail-safe](./0049-hypignore-usage-policy.spec.md#fail-safe)). + +The visible cost is that a **nested loosening does not cross spellings**: a +carve-out has to be declared in the same spelling as the entry it carves out of. +`hyp policy show` reports the class actually in force, so it is diagnosable. + +Specificity is measured on the **folded** spelling, not the declared one. NFD is +longer in code units than NFC for the same name, so a declared-string depth can +rank a decomposed ancestor above a composed descendant and invert +nearest-governs. + +### Cost + +The per-`cwd` memo is keyed on the **lexical** path and consulted **before** any +folding, so a cache hit costs exactly what it did before. Entry spellings and +the per-volume case verdict are computed once per list parse, inside the TTL +window LLP 0049 R6 already bounds. + +`String.prototype.normalize('NFC')` is roughly 60 ns on a pure-ASCII path, but +that is the case where the fold does nothing, so it is the wrong number to plan +against. When the string is **already** composed, normalize verifies and +returns: about 60 ns for a short ASCII path and about 90 ns for a 114-character +path with accented segments. When it is genuinely decomposed, which is exactly +the macOS case this section exists for, it has to recompose: about **550 ns** +for a 134-character NFD path, roughly 9x the ASCII figure, and it scales with +length (about 9.6 us for a pathological 1000-character all-decomposed path). + +That is affordable only because of where it sits. Folding is on the cache-**miss** +path (once per `cwd` per TTL window) and on the list parse (once per entry per +window), never per exchange. A miss over a 20-entry list with a long NFD `cwd` +measures about 8.2 us before this change and 9.7 us after. A caller that ever +moves the fold onto a per-row or per-exchange path has to re-measure with a +decomposed path, not an ASCII one. + +### Relationship to the symlink class + +This is the same shape [§canonicalization](#canonicalization) arrives at for +symlinks, for the same reason, and the two were found by the same review. They +are independent in *what* they fold - `realpath` cannot fold case or +normalization, and folding cannot resolve a symlink - but not in *how* they are +guarded, so they landed as one mechanism rather than two. + +The composition is a `map`, not a second set: `realpath` yields a **set** of +spellings, the fold is a **function** on a spelling, so the widened set is the +fold's image of the canonical set (`listScope` in `matcher.js`). And there is +exactly **one** two-pass argmax guard, evaluated over declared-and-unfolded +versus widened-and-folded, because the displacement hazard is identical whichever +mechanism gave a carve-out its extra reach. Running the guard twice would buy +nothing; running it once over the composed set is what makes +`class = max(pre_widening, widened)` hold for both mechanisms at once. + +### Not covered + +The fold is applied at the **gate** (`resolve` / list membership) and nowhere +else. The one-shot CLI membership sites (`hyp ignore --check`, `policy show`, +`policy unset`) and `hyp purge --subtree` now route through the shared +spelling-aware predicates [§canonicalization](#canonicalization) introduced +(`scopeGoverns`, `sameDirectory`, `governingListEntry`), so they fold **symlinks** +with the gate, but those predicates deliberately stop short of `foldPath` +(`canonicalScope` in `matcher.js`, as against `listScope`). On a case-insensitive +or NFD-carrying volume they can therefore still disagree with the gate about +normalization and case. + +That is the "widening is not free" rule above, not an oversight: the single +shared predicate is now the right *place* to add the fold, and adding it is still +gated on a per-volume normalization-insensitivity probe (or a darwin-only guard) +that does not exist yet. The gap below is stated in terms of that. + +The disagreement is bounded in one direction and not in the other, and the +difference matters enough to name each site: + +- **The CLI can never promise more protection than the gate delivers.** The + lexical predicate matches a subset of what the folded one does, and the gate's + class is `max(declared, folded)`, so any entry the CLI finds the gate also + found. There is no spelling on which `--check` reports a directory protected + while the gate forwards it. +- **`hyp ignore --check` / `policy show` report the right class and can name the + wrong scope.** The class comes from `resolve()`, so it is folded and correct. + Only the "which listed directory governs this?" lookup (`governingListEntry`) + is unfolded, so when the entry reaches `cwd` only by folding it falls back to + the queried path. The class is right; the governing directory shown, and the + residual row count scoped to it, are narrower than the truth. Note that the + row count makes this a *disclosure* predicate, which is why it does not simply + inherit the gate's fold either. +- **`policy unset` / `unignore --local-only` can refuse to remove an entry the + gate is enforcing**, when the user spells the path the other way. It reports + "not governed" and exits 0. That fails toward privacy: the opt-out stays on. +- **`hyp purge ` (the subtree target) silently retains rows it was asked + to delete**, when the rows were recorded under a different spelling of the + target. This is the one site that fails **away** from privacy: the user asked + for data to be deleted, the command reports success, and the rows remain. + Observed end to end (rows recorded NFD, purge argument NFC, and the reverse; + also a case alias): + + ``` + # the argument is typed NFC; the rows were recorded under the NFD spelling. + # the two render identically, which is the whole problem. + $ hyp purge ~/café/proj --yes + purged 0 rows from 0 partitions + $ echo $? + 0 + ``` + + Nothing is written to stderr and the exit status is 0, so the outcome is + indistinguishable from "that directory had nothing cached". Note the + inversion: a purge that *succeeds* prints the resurrection warning on stderr, + so the failing case is the **quieter** of the two. It is unchanged from the + pre-fold behaviour rather than introduced here, and it is tracked separately; + it is the reason this seam should not stay open for long. + +- **`hyp purge --ignored` is already covered by this change**, because that + target classifies each row through `resolver.resolve(row.cwd)` rather than + through a lexical prefix test, so it inherits the fold. Verified against + `master`: with an `ignore` entry declared NFC and rows recorded NFD, `master` + purges 0 rows and leaves the row, and this branch purges it. So marking the + directory and running `hyp purge --ignored` is the durable workaround for the + subtree gap above until that gap is closed. + +Whoever closes the subtree gap should note that it is **not** a matter of +dropping `foldPath` into the predicate. Purge deletes, so widening the match is +not free the way it is at the gate (see "NFC unconditionally, and only because +this is the gate"): on a Linux volume, folding would delete cached rows for a +genuinely different sibling directory that differs only by normalization. The +fix needs the fold gated on the volume actually being normalization-insensitive. +`scopeGoverns` - which already reroutes this purge call site for the symlink +class - is the right place to put it, and `canonicalScope` is the one line that +has to change once such a probe exists. + ## Consequences - Code that lands this carries `@ref LLP 0050 [implements]` on the adapter diff --git a/src/core/usage-policy/canonical.js b/src/core/usage-policy/canonical.js index c14659c0..6ebe6506 100644 --- a/src/core/usage-policy/canonical.js +++ b/src/core/usage-policy/canonical.js @@ -1,12 +1,16 @@ // @ts-check -import { createHash } from 'node:crypto' import nodeFs from 'node:fs' import path from 'node:path' import { Attr } from '../observability/attrs.js' import { getLogger } from '../observability/logger.js' +// One path digest for the whole usage-policy seam, so a hash in a +// `canonicalize_failed` line and a hash in a `case_probe_skipped` line name the +// same path when they name the same path. +import { hashPath } from './fold.js' + /** * `error_kind` for a `realpath(2)` that could not fully canonicalize a path. * Never fatal: the caller keeps the lexical spelling and the gate stays at @@ -14,19 +18,6 @@ import { getLogger } from '../observability/logger.js' */ export const PATH_CANONICALIZE_ERROR_KIND = 'path_canonicalize_failed' -/** - * Short one-way digest of a path, so a canonicalization failure is diagnosable - * (which path, how often, which errno) without dev telemetry ever carrying a - * raw local path. Same discipline as the `usage_policy.export_drop` aggregate - * in `src/core/cache/storage.js`. - * - * @param {string} p - * @returns {string} - */ -function hashPath(p) { - return createHash('sha256').update(p).digest('hex').slice(0, 16) -} - /** * The `errno` code of a filesystem error, as a lowercase token suitable for a * log attribute (`enoent`, `eacces`, `eloop`), or `unknown`. diff --git a/src/core/usage-policy/fold.js b/src/core/usage-policy/fold.js new file mode 100644 index 00000000..f2b7e306 --- /dev/null +++ b/src/core/usage-policy/fold.js @@ -0,0 +1,232 @@ +// @ts-check + +import { createHash } from 'node:crypto' +import nodeFs from 'node:fs' +import path from 'node:path' + +import { Attr } from '../observability/attrs.js' +import { getLogger } from '../observability/logger.js' + +/** + * `error_kind` for a per-volume case-sensitivity probe that could not reach a + * definite answer. Never fatal: an undetermined volume is treated as + * case-sensitive, which is exactly the pre-fold behaviour, so a failed probe + * can only lose reach the fold would have added. + */ +export const PATH_CASE_PROBE_ERROR_KIND = 'path_case_probe_failed' + +/** + * Short one-way digest of a path, so a fold decision or a skipped probe is + * diagnosable (which path, how often, which errno) without dev telemetry ever + * carrying a raw local path. Same discipline as the `usage_policy.export_drop` + * aggregate in `src/core/cache/storage.js`. + * + * @param {string} p + * @returns {string} + */ +export function hashPath(p) { + return createHash('sha256').update(p).digest('hex').slice(0, 16) +} + +/** + * The `errno` code of a filesystem error, as a lowercase token suitable for a + * log attribute (`enoent`, `eacces`, `eperm`), or `unknown`. + * + * @param {unknown} err + * @returns {string} + */ +function errnoOf(err) { + const code = /** @type {{ code?: unknown }} */ (err)?.code + return typeof code === 'string' && code !== '' ? code.toLowerCase() : 'unknown' +} + +/** + * Fold a path into the form two spellings of the *same* directory share. + * + * `realpath(2)` folds symlinks and nothing else. A filesystem can give one + * directory several spellings by two further mechanisms, and neither is + * reachable through `realpath`: + * + * 1. **Unicode normalization.** macOS frameworks and Finder-derived paths emit + * NFD (`e` + U+0301) while typed and JSON-transported paths are usually NFC + * (U+00E9). On a default APFS volume both `stat` and `chdir` to the same + * directory. NFC is applied **unconditionally** here: it is a total function + * that needs no filesystem access and cannot fail, and on a path that is + * already NFC (every path on a Linux box that was never typed on a Mac) it + * is the identity, so folding costs a comparison and changes nothing. + * 2. **Case.** On a case-insensitive volume `Proj` and `proj` are one + * directory. This is a property of the **volume**, not of the platform: an + * APFS volume can be formatted case-sensitive, and every ext4 volume is. So + * case is folded only when the caller passes a verdict for the volume the + * path lives on. Folding it unconditionally would merge two genuinely + * different directories on a case-sensitive volume, which is a correctness + * bug in the other direction. + * + * **Separator-preserving, and that is load-bearing.** The only consumer is a + * path-segment prefix test, so the fold has to distribute over `/`: + * `fold(a + '/' + b) === fold(a) + '/' + fold(b)`. It does. `/` is a starter + * with combining class 0 that participates in no canonical composition, so NFC + * is computed independently on each side of it, and `toLowerCase` maps `/` to + * itself. A fold that did not distribute could turn a non-descendant into a + * descendant across a segment boundary, so the property is asserted in the + * test suite rather than left to inspection. + * + * @ref LLP 0050#normalization [implements]: the fold that makes two spellings of one directory compare equal + * @param {string} p absolute path (already `path.resolve`d) + * @param {{ caseInsensitive?: boolean }} [opts] + * @returns {string} + */ +export function foldPath(p, { caseInsensitive = false } = {}) { + const nfc = p.normalize('NFC') + return caseInsensitive ? nfc.toLowerCase() : nfc +} + +/** + * A spelling of `p` whose **last segment** has the case of every cased + * character flipped, or `null` when that segment has no cased character (so no + * probe is possible from this path). + * + * Only the last segment is flipped, because the parent directories have to stay + * traversable under their exact spelling for the probe to be a statement about + * the volume `p` sits on rather than about every volume between it and the + * root. Flipping *every* cased character of that segment rather than one is + * deliberate: it is the spelling least likely to collide with a genuinely + * different sibling on a case-sensitive volume, and even a collision is caught, + * because what decides the verdict is the `dev`/`ino` identity of the two + * spellings, not whether the flipped name resolves. + * + * @param {string} p + * @returns {string | null} + */ +function flipCase(p) { + const cut = p.lastIndexOf(path.sep) + const head = cut < 0 ? '' : p.slice(0, cut + 1) + const tail = cut < 0 ? p : p.slice(cut + 1) + let flipped = '' + let anyFlipped = false + for (const ch of tail) { + const lower = ch.toLowerCase() + const upper = ch.toUpperCase() + if (lower !== upper) { + flipped += ch === lower ? upper : lower + anyFlipped = true + } else { + flipped += ch + } + } + return anyFlipped ? head + flipped : null +} + +/** + * Create a memoized per-volume case-sensitivity probe. + * + * The verdict is a property of the mounted volume, so it is memoized by the + * volume's `dev` number rather than by path. The memo is keyed on `dev`, which + * has to be *learned* before it can be consulted, so the cost is not zero on a + * hit: every call `stat`s `dir` itself (one `stat` per directory), and only the + * second, case-flipped `stat` is saved by the memo. So a list of `n` entries + * costs `n` stats plus one extra per distinct volume, per TTL window, rather + * than `2n`. That is still within the per-`cwd`-per-window bound LLP 0049 R6 + * sets for the ancestor walk, which already stats every ancestor. A volume + * cannot change its case-sensitivity without being unmounted and reformatted, + * at which point its `dev` changes too, so there is nothing for a TTL to + * refresh, and the flipped-spelling probe genuinely runs once per volume for + * the life of the resolver. + * + * **Inert off darwin.** On any other platform the probe returns `false` + * immediately and issues **no syscall at all**, because no shipping + * Linux/Windows filesystem this codebase targets presents the macOS + * case-insensitive-by-default behaviour that motivates the fold. That also + * means the whole probe is dead code on a Linux host, and therefore that its + * darwin behaviour cannot be executed, let alone verified, there. + * + * **Undetermined resolves to `false`**, which is the pre-fold behaviour: a + * directory that does not exist, a `stat` that is refused, or a path with no + * cased character all fold NFC only. A failed probe can therefore only fail to + * *add* reach; it can never remove a verdict some spelling already produced. + * + * @ref LLP 0050#normalization [implements]: case folding is per-volume and probed, never a platform constant + * @ref LLP 0049#fail-safe [constrained-by]: an undetermined probe resolves to the pre-fold behaviour, never to a looser gate + * @param {object} [deps] + * @param {string} [deps.platform] defaults to `process.platform` + * @param {(p: string) => { dev: number, ino: number }} [deps.statSync] + * @param {(name: string, fields?: Record) => void} [deps.logSkip] + * @returns {(dir: string) => boolean} + */ +export function createVolumeCaseProbe({ + platform = process.platform, + statSync = nodeFs.statSync, + logSkip, +} = {}) { + if (platform !== 'darwin') return () => false + + /** @type {Map} */ + const byDev = new Map() + + /** + * @param {string} dir + * @param {string} reason + * @param {string} errno + */ + function skip(dir, reason, errno) { + const emit = logSkip ?? defaultLogSkip + emit('usage_policy.case_probe_skipped', { + [Attr.COMPONENT]: 'usage-policy', + [Attr.OPERATION]: 'case_probe', + [Attr.STATUS]: 'skipped', + [Attr.ERROR_KIND]: PATH_CASE_PROBE_ERROR_KIND, + reason, + errno, + path_hash: hashPath(dir), + }) + } + + return function probe(dir) { + /** @type {{ dev: number, ino: number }} */ + let st + try { + st = statSync(dir) + } catch (err) { + skip(dir, 'stat_failed', errnoOf(err)) + return false + } + const memoized = byDev.get(st.dev) + if (memoized !== undefined) return memoized + + const flipped = flipCase(dir) + if (flipped === null) { + // Not memoized: another path on this same volume may well have a cased + // character, and would then reach a definite answer. + skip(dir, 'no_cased_character', 'none') + return false + } + let verdict + try { + const other = statSync(flipped) + verdict = other.dev === st.dev && other.ino === st.ino + } catch (err) { + // `ENOENT` here is the *informative* outcome: the flipped spelling does + // not resolve, so the volume is case-sensitive. Any other errno is a + // genuinely undetermined probe and is not memoized. + const errno = errnoOf(err) + if (errno !== 'enoent') { + skip(dir, 'stat_failed', errno) + return false + } + verdict = false + } + byDev.set(st.dev, verdict) + return verdict + } +} + +/** + * @param {string} name + * @param {Record} [fields] + * @returns {void} + */ +function defaultLogSkip(name, fields) { + // A directory that does not exist is routine at this seam (a deleted `cwd`, + // a mark for a not-yet-created directory), so this is never a warning. + getLogger('usage-policy').debug(name, fields) +} diff --git a/src/core/usage-policy/index.js b/src/core/usage-policy/index.js index 089d4b32..140485a2 100644 --- a/src/core/usage-policy/index.js +++ b/src/core/usage-policy/index.js @@ -12,11 +12,17 @@ export { sameDirectory, scopeGoverns, } from './matcher.js' -// Symlink canonicalization for the gate (LLP 0050 #canonicalization): a -// directory is matched over the set of spellings that denote it, so a -// `.hypignore` or a machine-local entry cannot be escaped (or lost) by reaching -// the same directory through a symlink. +// The two mechanisms by which one directory has several spellings, and the gate +// folds both. Symlink canonicalization (LLP 0050 #canonicalization): a directory +// is matched over the set of spellings that denote it, so a `.hypignore` or a +// machine-local entry cannot be escaped (or lost) by reaching the same directory +// through a symlink. export { canonicalizeDirSync, canonicalSpellings, PATH_CANONICALIZE_ERROR_KIND } from './canonical.js' +// And the spelling fold applied to each of those (LLP 0050 #normalization): +// Unicode-NFC always, case only on a volume probed case-insensitive. Exported so +// a caller that has to agree with the gate's verdict folds by the same rule +// instead of growing a second one. +export { createVolumeCaseProbe, foldPath, PATH_CASE_PROBE_ERROR_KIND } from './fold.js' // The terminal capture-seam drop sentinel (LLP 0050): an adapter projector // returns it for an `.hypignore`-ignored exchange, and the gateway dispatcher // stops on it (never falls through to a later projector) and logs it as a drop. diff --git a/src/core/usage-policy/matcher.js b/src/core/usage-policy/matcher.js index 2e209ded..6c16dd74 100644 --- a/src/core/usage-policy/matcher.js +++ b/src/core/usage-policy/matcher.js @@ -3,12 +3,16 @@ import nodeFs from 'node:fs' import path from 'node:path' +import { Attr } from '../observability/attrs.js' +import { getLogger } from '../observability/logger.js' + import { canonicalSpellings } from './canonical.js' +import { createVolumeCaseProbe, foldPath, hashPath } from './fold.js' import { parseHypignore } from './format.js' import { LocalOnlyListUnreadableError } from './local_only.js' /** - * @import { LocalOnlyEntry, ResolveResult, UsagePolicyResolver } from '../../../src/core/usage-policy/types.js' + * @import { ListScope, LocalOnlyEntry, ResolveResult, UsageClass, UsagePolicyResolver } from '../../../src/core/usage-policy/types.js' */ const HYPIGNORE_FILENAME = '.hypignore' @@ -73,9 +77,15 @@ const LOCAL_ONLY_LIST_VERSION_V2 = 2 * exists, the TTL is the leak bound. * * Both sides of every comparison are resolved over the *set* of spellings that - * denote a directory (as-given/lexical plus canonical), never over one chosen - * spelling: see {@link canonicalSpellings} for why the set, not the canonical - * form alone, is the privacy-preserving choice. + * denote a directory, never over one chosen spelling. A filesystem hands out + * those spellings by two independent mechanisms, and the resolver folds both: + * symlinks, resolved by {@link canonicalSpellings} into a set of strings (see + * there for why the set, not the canonical form alone, is the privacy-preserving + * choice), and Unicode normalization plus per-volume case, resolved by + * {@link foldPath} into one folded image of each of those strings. Neither can + * remove a spelling the other produced, and the merge across spellings takes the + * most restrictive verdict, so the composition can only ever move the gate + * toward more restrictive. * * fs, the clock, and the TTL are injected for tests; fs defaults to `node:fs`, * the clock to `Date.now`, and the TTL to `CACHE_TTL_MS`. @@ -86,6 +96,7 @@ const LOCAL_ONLY_LIST_VERSION_V2 = 2 * @ref LLP 0052#matcher [implements]: bounded-TTL staleness so a mid-run .hypignore is honored without a daemon restart * @ref LLP 0070#resolver [implements]: one shared resolver, two sources, most-restrictive class wins * @ref LLP 0071 [implements]: the machine-local list is the second source + * @ref LLP 0050#normalization [implements]: list membership compares folded spellings, so an NFC/NFD divergence between two processes does not open the gate * @param {object} [deps] * @param {(path: string, encoding: 'utf8') => string} [deps.readFileSync] * @param {(path: string) => boolean} [deps.existsSync] @@ -97,6 +108,12 @@ const LOCAL_ONLY_LIST_VERSION_V2 = 2 * @param {string} [deps.localOnlyListPath] absolute path of the machine-local * `local-only` list (`localOnlyListPath(stateDir)`, LLP 0071); omitted => * the resolver behaves exactly as it did before the list existed + * @param {(dir: string) => boolean} [deps.caseInsensitiveVolume] per-volume + * case-sensitivity verdict for an entry's directory; defaults to + * {@link createVolumeCaseProbe}, which is inert (constant `false`, no + * syscall) off darwin. Injected so the folding logic can be exercised for a + * case-insensitive volume on a host that has none + * @param {(name: string, fields?: Record) => void} [deps.logEvent] * @returns {UsagePolicyResolver} */ export function createUsagePolicyResolver({ @@ -106,20 +123,25 @@ export function createUsagePolicyResolver({ now = Date.now, ttlMs = CACHE_TTL_MS, localOnlyListPath, + caseInsensitiveVolume, + logEvent, } = {}) { /** @type {Map} */ const cache = new Map() - /** @type {{ scopes: { entry: LocalOnlyEntry, spellings: string[] }[], expiresAt: number } | null} */ + /** @type {{ scopes: ListScope[], expiresAt: number } | null} */ let listCache = null + const emit = logEvent ?? emitDebug + const probeVolume = caseInsensitiveVolume ?? createVolumeCaseProbe({ logSkip: emit }) /** * Resolve `cwd` over every spelling that denotes it, returning the most * restrictive verdict any spelling produces. * - * The cache is keyed on the *lexical* path, so a hit costs no `realpath` at - * all: the canonicalization syscall happens once per distinct `cwd` per TTL - * window, the same bound LLP 0049 R6 already sets for the ancestor walk, not - * once per recorded exchange. + * The cache is keyed on the *lexical* path and consulted before any + * canonicalization or folding, so a hit costs exactly what it did before + * either existed: the `realpath` syscall and the fold happen once per distinct + * `cwd` per TTL window, the same bound LLP 0049 R6 already sets for the + * ancestor walk, not once per recorded exchange. * * @param {string} cwd * @returns {ResolveResult} @@ -197,22 +219,25 @@ export function createUsagePolicyResolver({ * by the more restrictive class. * * An entry governs `cwd` when *any* spelling of the entry's directory - * equals-or-contains `cwd`, so an entry declared by a symlink spelling still - * governs the real directory and vice versa. Specificity is measured on the - * spelling that actually matched, so nested entries still resolve - * nearest-governs regardless of which spelling each was declared with. - * Widening an entry's reach must not *loosen* the list, which is what - * {@link selectGoverning} guarantees. + * equals-or-contains `cwd`: symlink-resolved as well as declared, and each of + * those folded (Unicode-normalized, and case-folded on a volume probed + * case-insensitive). So an entry declared by a symlink spelling still governs + * the real directory and vice versa, and an entry declared NFC still governs a + * `cwd` that arrives NFD. Specificity is measured on the spelling that + * actually matched, so nested entries still resolve nearest-governs regardless + * of which spelling each was declared with. Widening an entry's reach must not + * *loosen* the list, which is what {@link selectGoverning} guarantees. * * @ref LLP 0071 [implements]: segment-aware equal-or-descendant list membership, second resolver source * @ref LLP 0050#canonicalization [implements]: an entry governs through any spelling of its declared directory + * @ref LLP 0050#normalization [implements]: an entry governs through any spelling the volume folds together * @ref LLP 0103 [implements]: the entry's own class governs, not a hardcoded `local-only` * @param {string} cwd absolute, already `path.resolve`d * @param {number} at current clock reading (ms) * @returns {ResolveResult | null} `null` when nothing in the list governs `cwd` */ function matchList(cwd, at) { - const governing = selectGoverning(cwd, getListScopes(at)) + const governing = selectGoverning(cwd, getListScopes(at), reportFold) if (governing === null) return null return { class: governing.entry.class, @@ -221,21 +246,42 @@ export function createUsagePolicyResolver({ } } + /** + * Structured signal for the one interesting outcome: the widened pass reached + * a **more restrictive** verdict than the declared spellings did, i.e. a + * spelling divergence that would otherwise have opened the gate. Paths are + * hashed, never logged raw, the same discipline as the + * `usage_policy.export_drop` aggregate. + * + * @param {string} cwd + * @param {UsageClass | null} declaredClass + * @param {UsageClass} foldedClass + * @returns {void} + */ + function reportFold(cwd, declaredClass, foldedClass) { + emit('usage_policy.fold_tightened', { + [Attr.COMPONENT]: 'usage-policy', + [Attr.OPERATION]: 'match_list', + [Attr.STATUS]: 'ok', + declared_class: declaredClass ?? 'none', + folded_class: foldedClass, + cwd_hash: hashPath(cwd), + }) + } + /** * The list entries paired with every spelling of each entry's declared - * directory, computed once per TTL window along with the parse, so resolving - * many `cwd`s in one window costs one `realpath` per entry rather than one per - * entry per `cwd`. + * directory (symlink-resolved, then folded) and the case verdict for the + * volume it lives on, computed once per TTL window along with the parse. So + * resolving many `cwd`s in one window costs one `realpath` and one fold per + * entry rather than one per entry per `cwd`. * * @param {number} at - * @returns {{ entry: LocalOnlyEntry, spellings: string[] }[]} + * @returns {ListScope[]} */ function getListScopes(at) { if (listCache && listCache.expiresAt > at) return listCache.scopes - const scopes = readListEntriesSync().map((entry) => ({ - entry, - spellings: canonicalSpellings(entry.dir, { realpathSync }), - })) + const scopes = readListEntriesSync().map((entry) => listScope(entry, probeVolume, { realpathSync })) listCache = { scopes, expiresAt: at + ttlMs } return scopes } @@ -306,6 +352,19 @@ export function createUsagePolicyResolver({ return { resolve, isIgnored } } +/** + * Default sink for the resolver's structured signals. Both of them (a fold that + * tightened a verdict, a case probe that could not reach an answer) are routine + * rather than faults, so neither is ever louder than `debug`. + * + * @param {string} name + * @param {Record} [fields] + * @returns {void} + */ +function emitDebug(name, fields) { + getLogger('usage-policy').debug(name, fields) +} + /** * True when `cwd` equals `dir`, or is a path-segment descendant of it. * Segment-aware: `/a/bc` is not a descendant of `/a/b` even though it shares @@ -329,53 +388,143 @@ export function isEqualOrDescendant(cwd, dir) { } /** - * Length of the spelling in `dirSpellings` that equals-or-contains `cwd`, or - * `null` when none does. The length stands in for specificity, the same way the - * `.hypignore` walk's nearest-governs rule does; measuring it on the *matched* - * spelling keeps nested entries ordered correctly even when they were declared - * with different spellings of the same tree. - * - * Returning the first match rather than the longest is not a shortcut: two - * spellings of one directory can only both contain the same `cwd` if one is a - * lexical ancestor of the other, and the canonical form can never be a strict - * lexical descendant of the as-given form (that would need a symlink pointing - * inside itself, which `realpath` reports as `ELOOP`). So when both match, the - * as-given spelling - which `canonicalSpellings` puts first - is the longer one. + * The scope one machine-local entry occupies at the **gate**: its declared + * spelling, the widened set (symlink-resolved *and* folded), and the case + * verdict the `cwd` side has to be folded through to match it. + * + * The two widenings compose in exactly one direction, which is why the widened + * set is a `map` and not a second set: `realpath` produces a *set* of strings + * (as-given, canonical), while the fold is a *function* on a string, so the + * widened set is the fold's image of the canonical set. Computing it here means + * an entry is canonicalized and folded once per TTL window, not once per `cwd`. + * + * @ref LLP 0050#canonicalization [implements]: the entry side is a set of spellings, not one chosen string + * @ref LLP 0050#normalization [implements]: each of those spellings is compared through the volume's fold + * @param {LocalOnlyEntry} entry + * @param {(dir: string) => boolean} probeVolume + * @param {{ realpathSync?: (p: string) => string, component?: string }} deps + * @returns {ListScope} + */ +function listScope(entry, probeVolume, deps) { + const spellings = canonicalSpellings(entry.dir, deps) + const caseInsensitive = probeVolume(entry.dir) + return { + entry, + folded: true, + caseInsensitive, + declaredSpellings: [spellings[0]], + widenedSpellings: spellings.map((spelling) => foldPath(spelling, { caseInsensitive })), + } +} + +/** + * The same scope for a **one-shot CLI** call site: symlink-widened, deliberately + * *not* folded. + * + * The asymmetry is the point, and it is a decision rather than an oversight. + * Unconditional NFC folding is sound at the gate only because the gate's answer + * is `max(declared, widened)` on the restrictiveness lattice, so a fold that + * merges two genuinely distinct directories (which it does on Linux, where NFC + * and NFD names are two inodes) can only ever over-suppress. That argument does + * not survive the trip to a CLI verb: `hyp purge --subtree` **deletes** through + * {@link scopeGoverns}, `policy unset` **removes an opt-out** through it, and + * `sameDirectory` decides which stored declaration to **replace**. In each of + * those, widening the match destroys something the user did not name. Closing + * that gap needs a per-volume normalization-insensitivity probe (the existing + * probe answers only the case question) or a darwin-only guard, which is tracked + * in the LLP rather than smuggled in here. + * + * @ref LLP 0050#normalization [constrained-by]: "do not reuse foldPath in a predicate where widening is not free" + * @param {LocalOnlyEntry} entry + * @param {{ realpathSync?: (p: string) => string, component?: string }} deps + * @returns {ListScope} + */ +function canonicalScope(entry, deps) { + const spellings = canonicalSpellings(entry.dir, deps) + return { + entry, + folded: false, + caseInsensitive: false, + declaredSpellings: [spellings[0]], + widenedSpellings: spellings, + } +} + +/** + * Length of the longest spelling in `dirSpellings` that equals-or-contains + * `cwd`, or `null` when none does. The length stands in for specificity, the + * same way the `.hypignore` walk's nearest-governs rule does; measuring it on + * the *matched* spelling keeps nested entries ordered correctly even when they + * were declared with different spellings of the same tree. + * + * The longest rather than the first, deliberately. Over symlink spellings alone + * the two coincide, because two spellings of one directory can only both contain + * the same `cwd` if one is a lexical ancestor of the other, the canonical form + * can never be a strict lexical descendant of the as-given one (that would need + * a symlink pointing inside itself, which `realpath` reports as `ELOOP`), and + * `canonicalSpellings` puts the as-given form first. Folding breaks that: NFC is + * length-reducing, so the folded image of the as-given spelling is not + * necessarily the longer one any more. Taking the maximum makes the ordering + * independent of how the spellings happen to be arranged. * * @param {string} cwd absolute, already `path.resolve`d - * @param {readonly string[]} dirSpellings ordered as-given first + * @param {readonly string[]} dirSpellings folded consistently with `cwd` * @returns {number | null} */ function matchDepth(cwd, dirSpellings) { + /** @type {number | null} */ + let depth = null for (const dir of dirSpellings) { - if (isEqualOrDescendant(cwd, dir)) return dir.length + if (isEqualOrDescendant(cwd, dir) && (depth === null || dir.length > depth)) depth = dir.length } - return null + return depth } /** * The nearest-governs winner over `scopes`: the entry whose matched spelling is - * the longest, ties broken by the more restrictive class. `spellingLimit` caps - * how many of each entry's spellings may match, so the same rule can be run - * over the declared spellings alone or over the widened set. + * the longest, ties broken by the more restrictive class. + * + * When `widened` is false this compares `cwd` against each entry's declared + * spelling alone, unfolded, which is bit-for-bit the rule the matcher applied + * before either widening existed. When it is true it compares `cwd` against the + * scope's widened set, folding `cwd` through the same verdict that set was built + * with, so an entry reaches every spelling the filesystem treats as the same + * directory: a symlink always, and a Unicode or case respelling for a scope that + * opted into the fold ({@link listScope} does, {@link canonicalScope} does not). * * @param {string} cwd absolute, already `path.resolve`d - * @param {readonly { entry: LocalOnlyEntry, spellings: readonly string[] }[]} scopes - * @param {number} spellingLimit + * @param {readonly ListScope[]} scopes + * @param {boolean} widened * @returns {{ entry: LocalOnlyEntry, depth: number } | null} */ -function deepestMatch(cwd, scopes, spellingLimit) { +function deepestMatch(cwd, scopes, widened) { + // `foldPath(cwd, { caseInsensitive: true })` is `foldPath(cwd)` lowered, so + // each variant is computed at most once per call rather than once per entry, + // and not at all for a scope set that does not fold. + /** @type {string | null} */ + let nfcCwd = null + /** @type {string | null} */ + let loweredCwd = null /** @type {{ entry: LocalOnlyEntry, depth: number } | null} */ let best = null - for (const { entry, spellings } of scopes) { - const depth = matchDepth(cwd, spellingLimit >= spellings.length ? spellings : spellings.slice(0, spellingLimit)) + for (const scope of scopes) { + let target = cwd + let dirs = scope.declaredSpellings + if (widened) { + dirs = scope.widenedSpellings + if (scope.folded) { + nfcCwd ??= foldPath(cwd) + target = scope.caseInsensitive ? (loweredCwd ??= nfcCwd.toLowerCase()) : nfcCwd + } + } + const depth = matchDepth(target, dirs) if (depth === null) continue if ( best === null || depth > best.depth || - (depth === best.depth && CLASS_RANK[entry.class] > CLASS_RANK[best.entry.class]) + (depth === best.depth && CLASS_RANK[scope.entry.class] > CLASS_RANK[best.entry.class]) ) { - best = { entry, depth } + best = { entry: scope.entry, depth } } } return best @@ -384,44 +533,57 @@ function deepestMatch(cwd, scopes, spellingLimit) { /** * The machine-local entry that governs `cwd`, over precomputed spellings. * - * Nearest-governs alone is *not* monotone in the set of spellings, which is the - * one place canonicalization could have made the gate **less** restrictive than - * the lexical matcher it replaced. An explicit `full` (or merely less - * restrictive) entry that gains reach through its canonical spelling can become - * the deepest match and so displace a broader restrictive entry that already - * governed: a carve-out declared under one spelling would punch a hole in a - * private tree declared under the other, and the directory would start - * recording and forwarding. Nothing about "resolve over a set of spellings" - * prevents that on its own, because the argmax-over-depth step in the middle - * discards verdicts rather than merging them. - * - * So the rule is run twice - once over the declared spellings alone (exactly - * what the pre-canonicalization matcher decided) and once over the widened set + * Nearest-governs alone is *not* monotone in the set of spellings an entry can + * reach, and that is the one place widening the entry side could have made the + * gate **less** restrictive than the plain string matcher it replaced. An + * explicit `full` (or merely less restrictive) entry that gains reach through a + * widened spelling can become the deepest match and so displace a broader + * restrictive entry that already governed: a carve-out declared under one + * spelling would punch a hole in a private tree declared under the other, and + * the directory would start recording and forwarding. Nothing about "resolve + * over a set of spellings" prevents that on its own, because the + * argmax-over-depth step in the middle discards verdicts rather than merging + * them. + * + * This is one guard for both widenings, and it has to be, because they compose: + * an entry reaches `cwd` through a symlink (LLP 0050 §canonicalization), through + * a Unicode or case respelling (§normalization), or through both at once, and + * the displacement hazard is identical in each case. So the rule is run twice - + * once over the declared spellings alone, unfolded (exactly what the matcher + * decided before either widening existed) and once over the widened, folded set * - and the more restrictive of the two answers wins, the declared one breaking - * a class tie because it is the spelling the user typed. Widening an entry's - * reach can then only ever add restriction, never remove it, which is the - * fail-toward-privacy property LLP 0050 §canonicalization claims. + * a class tie because it is the spelling the user typed. The resolved class is + * therefore `max(pre_widening, widened)` on the restrictiveness lattice by + * construction: widening an entry's reach can only ever add restriction, never + * remove it, which is the fail-toward-privacy property both sections claim. The + * visible cost is that a nested loosening does not cross spellings, which is the + * direction LLP 0049 §fail-safe picks. * * Note the exact reach of the guard: it preserves a verdict the **declared** * pass produced, so it blocks a cross-spelling loosening only when the broader - * restrictive entry matches `cwd` by its own declared spelling. If that entry - * reaches `cwd` only through canonicalization, the declared pass matches + * restrictive entry matches `cwd` by its own declared, unfolded spelling. If + * that entry reaches `cwd` only through a widening, the declared pass matches * nothing and plain nearest-governs picks between entries that are all in the - * canonical namespace, so a deeper carve-out wins. That is still never a - * demotion: the lexical matcher matched neither entry in that shape either. + * widened namespace, so a deeper carve-out wins. That is still never a + * demotion: the pre-widening matcher matched neither entry in that shape either. * * @ref LLP 0050#canonicalization [implements]: canonicalization only ever moves the gate toward more restrictive, entry side included + * @ref LLP 0050#normalization [implements]: a folded spelling only ever adds restriction, entry side included * @ref LLP 0049#fail-safe [constrained-by]: a widened reach must resolve to "suppress more", never to "starts forwarding" * @param {string} cwd absolute, already `path.resolve`d - * @param {readonly { entry: LocalOnlyEntry, spellings: readonly string[] }[]} scopes + * @param {readonly ListScope[]} scopes + * @param {(cwd: string, declaredClass: UsageClass | null, widenedClass: UsageClass) => void} [onTightened] * @returns {{ entry: LocalOnlyEntry, depth: number } | null} */ -function selectGoverning(cwd, scopes) { - const asDeclared = deepestMatch(cwd, scopes, 1) - const widened = deepestMatch(cwd, scopes, Number.POSITIVE_INFINITY) - if (asDeclared === null) return widened - if (widened === null) return asDeclared - return CLASS_RANK[widened.entry.class] > CLASS_RANK[asDeclared.entry.class] ? widened : asDeclared +function selectGoverning(cwd, scopes, onTightened) { + const asDeclared = deepestMatch(cwd, scopes, false) + const widened = deepestMatch(cwd, scopes, true) + const declaredRank = asDeclared === null ? CLASS_RANK.full : CLASS_RANK[asDeclared.entry.class] + if (widened !== null && CLASS_RANK[widened.entry.class] > declaredRank) { + if (onTightened) onTightened(cwd, asDeclared === null ? null : asDeclared.entry.class, widened.entry.class) + return widened + } + return asDeclared ?? widened } /** @@ -436,6 +598,9 @@ function selectGoverning(cwd, scopes) { * how `--check` / `policy show` ends up naming an entry the gate did not use * (R8: one shared thing, not a second copy of the selection rule). * + * Symlink-widened but **not folded**, unlike the gate: see + * {@link canonicalScope} for why the fold stops at the gate. + * * Does up to two `realpath` calls per entry plus two for `dir`, so it is for * one-shot CLI use, not a per-row loop. * @@ -447,7 +612,7 @@ function selectGoverning(cwd, scopes) { * @returns {LocalOnlyEntry | null} */ export function governingListEntry(dir, entries, deps = {}) { - const scopes = entries.map((entry) => ({ entry, spellings: canonicalSpellings(entry.dir, deps) })) + const scopes = entries.map((entry) => canonicalScope(entry, deps)) /** @type {{ entry: LocalOnlyEntry, depth: number } | null} */ let best = null for (const spelling of canonicalSpellings(dir, deps)) { @@ -470,11 +635,16 @@ export function governingListEntry(dir, entries, deps = {}) { * callers that are comparing two already-canonical strings and must not touch * the filesystem. * + * Symlink-widened but **not folded**: see {@link canonicalScope}. This one is + * the sharpest case, because `hyp purge --subtree` routes its *deletion* + * predicate through here. + * * Does up to two `realpath` calls, so callers on a per-row loop should memoize * per distinct path (`src/core/cache/purge.js` does). * * @ref LLP 0069#requirements [implements]: R8, one shared equal-or-descendant test, now spelling-agnostic * @ref LLP 0050#canonicalization [implements]: CLI membership answers agree with the gate's verdict + * @ref LLP 0050#normalization [constrained-by]: the fold stops at the gate; a deletion predicate must not widen for free * @param {string} cwd * @param {string} dir * @param {{ realpathSync?: (p: string) => string, component?: string }} [deps] @@ -491,7 +661,12 @@ export function scopeGoverns(cwd, dir, deps = {}) { * re-marking a directory through a different spelling updates its class rather * than adding a second governor for the same directory. * + * Symlink-widened but **not folded**: see {@link canonicalScope}. Widening this + * one merges two stored entries into one, which silently drops a class the user + * declared, so it is not free either. + * * @ref LLP 0050#canonicalization [implements]: entry identity is the directory, not the string + * @ref LLP 0050#normalization [constrained-by]: the fold stops at the gate; merging two declarations is not free * @param {string} a * @param {string} b * @param {{ realpathSync?: (p: string) => string, component?: string }} [deps] diff --git a/src/core/usage-policy/types.d.ts b/src/core/usage-policy/types.d.ts index 724d54d2..6a729a6a 100644 --- a/src/core/usage-policy/types.d.ts +++ b/src/core/usage-policy/types.d.ts @@ -61,6 +61,33 @@ export interface LocalOnlyEntry { class: UsageClass } +// A machine-local list entry paired with every spelling of its declared `dir` +// that the gate compares through, and the case-sensitivity verdict for the +// volume that directory lives on. Computed once per list parse per TTL window, +// so resolving many `cwd`s in one window canonicalizes and folds each entry +// once rather than once per `cwd`. +// +// The two spelling sets are what the matcher's two passes compare against. +// `declaredSpellings` is the single spelling the user declared, unfolded: the +// rule the matcher applied before either widening existed. `widenedSpellings` +// is `canonicalSpellings(dir)` (as-given plus symlink-resolved, LLP 0050 +// §canonicalization), folded when `folded` is set (LLP 0050 §normalization). +// +// `folded` is true for the gate's scopes and false for the one-shot CLI +// helpers', because unconditional NFC folding is sound only where widening is +// free, which is the gate and not a deletion or disclosure predicate. +// `caseInsensitive` is false on every non-darwin host, on any volume whose probe +// was undetermined, and on every unfolded scope; it is carried because the `cwd` +// side has to be folded through the *same* verdict for the comparison to mean +// anything. +export interface ListScope { + entry: LocalOnlyEntry + folded: boolean + caseInsensitive: boolean + declaredSpellings: string[] + widenedSpellings: string[] +} + // Version-2 on-disk shape of the machine-local list (LLP 0103): the // class-per-entry store that replaces the version-1 bare `dirs` array. export interface LocalOnlyListFileV2 { diff --git a/test/core/usage-policy-fold.test.js b/test/core/usage-policy-fold.test.js new file mode 100644 index 00000000..bd358cba --- /dev/null +++ b/test/core/usage-policy-fold.test.js @@ -0,0 +1,438 @@ +// @ts-check + +// Regression tests for the path-spelling fold: `realpath(2)` folds symlinks +// and nothing else, so on a filesystem that treats two spellings of one +// directory as the same directory the shared gate compared two strings that +// differ and returned `full` for a directory the user opted out of (#483). +// +// **What these tests can and cannot prove on a Linux host.** The Unicode half +// is honest here: `String.prototype.normalize` is a pure function of the +// string, the two spellings are built as literals, and the assertion is that +// the *comparison logic* folds them together. That is the whole mechanism, and +// it is the half that actually bites, because the two paths being compared are +// produced by different processes at different times (a CLI resolving a mark +// versus a client reporting a `cwd`), so an NFC/NFD divergence between them is +// ordinary rather than user error. +// +// The case half is different. Whether a *volume* folds `Proj` and `proj` is a +// property of the filesystem, and ext4 does not, so the tests below drive the +// case-folding logic through an **injected** volume verdict. That covers the +// matcher's behaviour given a verdict; it does not and cannot cover the probe +// that produces the verdict on macOS. The probe is asserted only to be inert +// off darwin, which is the one thing this host can witness. +// +// @ref LLP 0050#normalization [tests]: folding is only ever additive restriction, and NFC divergence no longer opens the gate + +import test from 'node:test' +import assert from 'node:assert/strict' + +import { createUsagePolicyResolver } from '../../src/core/usage-policy/matcher.js' +import { createVolumeCaseProbe, foldPath, PATH_CASE_PROBE_ERROR_KIND } from '../../src/core/usage-policy/fold.js' + +/** + * @import { UsageClass, UsagePolicyResolver } from '../../src/core/usage-policy/types.js' + */ + +// The same four characters, composed and decomposed. Spelled as \u escapes, so +// the source file is pure ASCII and an editor, a merge tool, or a `git` filter +// cannot silently re-normalize it. Re-normalizing raw literals would collapse +// both constants to the same string and make every test below pass vacuously; +// escaping makes that unrepresentable rather than merely detectable, and the +// tripwire immediately below still asserts the premise for anyone who +// reintroduces raw characters. +const NFC = 'caf\u00e9' +const NFD = 'cafe\u0301' + +// The premise the whole file rests on. If this ever fails, nothing after it +// means anything. +test('the two fixture spellings are genuinely different strings that NFC folds together', () => { + assert.notEqual(NFC, NFD) + assert.equal(NFD.normalize('NFC'), NFC) + assert.equal(NFC.normalize('NFC'), NFC) +}) + +const LIST = '/state/usage-policy/local-only.json' + +/** + * A resolver over an in-memory machine-local list. No `.hypignore` exists, so + * every verdict below comes from list membership, which is the comparison + * under test. + * + * @param {readonly { dir: string, class: UsageClass }[]} entries + * @param {{ caseInsensitiveVolume?: (dir: string) => boolean, logEvent?: (name: string, fields?: Record) => void }} [deps] + * @returns {UsagePolicyResolver} + */ +function resolverOver(entries, deps = {}) { + const files = { [LIST]: JSON.stringify({ version: 2, entries }) } + return createUsagePolicyResolver({ + existsSync: (p) => Object.prototype.hasOwnProperty.call(files, p), + readFileSync: (p) => /** @type {Record} */ (files)[p], + localOnlyListPath: LIST, + ...deps, + }) +} + +// --- the leak: NFC/NFD divergence between two processes ------------------- + +test('resolve: a local-only entry declared NFC still governs a cwd that arrives NFD', () => { + const r = resolverOver([{ dir: `/root/${NFC}/proj`, class: 'local-only' }]) + assert.equal(r.resolve(`/root/${NFD}/proj`).class, 'local-only') +}) + +test('resolve: an ignore entry declared NFD still governs a cwd that arrives NFC', () => { + const r = resolverOver([{ dir: `/root/${NFD}`, class: 'ignore' }]) + assert.equal(r.resolve(`/root/${NFC}/proj/sub`).class, 'ignore') + assert.equal(r.isIgnored(`/root/${NFC}/proj/sub`), true) +}) + +test('resolve: the fold is on the ancestor segment, not only the leaf', () => { + // The divergent segment is an ancestor of the `cwd`, so the prefix test has + // to survive folding across a `/` boundary. + const r = resolverOver([{ dir: `/root/${NFD}/a/b`, class: 'local-only' }]) + assert.equal(r.resolve(`/root/${NFC}/a/b/c/d`).class, 'local-only') +}) + +// --- the fold must not merge across a path-segment boundary --------------- + +test('foldPath distributes over the path separator, so a prefix test stays segment-aware', () => { + for (const p of [`/root/${NFD}/a`, `/root/${NFC}/a`, '/plain/ascii/path', '/']) { + const segments = p.split('/') + assert.equal(foldPath(p), segments.map((s) => foldPath(s)).join('/')) + assert.equal(foldPath(p, { caseInsensitive: true }), segments.map((s) => foldPath(s, { caseInsensitive: true })).join('/')) + } +}) + +test('resolve: a sibling whose name merely shares a folded prefix is still NOT matched', () => { + const r = resolverOver([{ dir: `/root/${NFC}`, class: 'ignore' }]) + assert.equal(r.resolve(`/root/${NFD}-other`).class, 'full') + assert.equal(r.resolve(`/root/${NFC}xyz`).class, 'full') +}) + +// --- the property that keeps this from becoming a forwarding leak --------- + +test('resolve: a carve-out that gains reach only by folding does not punch a hole in a broader restrictive entry', () => { + // The analogue of the regression PR #482's round-1 review found: an argmax + // over match depth discards verdicts, so a less restrictive entry that only + // matches once folded can become the deepest match and displace a broader + // restrictive entry that already governed. Without the two-pass rule this + // resolves to `full` and the directory starts recording and forwarding. + const r = resolverOver([ + { dir: '/root/real', class: 'ignore' }, + { dir: `/root/real/${NFC}`, class: 'full' }, + ]) + assert.equal(r.resolve(`/root/real/${NFD}/sub`).class, 'ignore') + assert.equal(r.resolve(`/root/real/${NFD}`).class, 'ignore') +}) + +test('resolve: an entry that gains reach by folding overrides a shallower explicit full marker', () => { + // The tightening direction, against an entry that already matched: LLP 0103's + // explicit `full` marker governs `/root`, and the restrictive entry only + // reaches `cwd` once folded. Nearest-governs then has to prefer the deeper + // folded match, or an opted-out subtree keeps forwarding under the marker its + // parent carries. + const r = resolverOver([ + { dir: '/root', class: 'full' }, + { dir: `/root/${NFD}`, class: 'ignore' }, + ]) + assert.equal(r.resolve('/root/elsewhere').class, 'full') + assert.equal(r.resolve(`/root/${NFC}/deep`).class, 'ignore') +}) + +test('resolve: a carve-out declared in the same spelling as the entry it carves out of is still honored', () => { + // The positive half: the two-pass rule must not over-restrict a legitimate + // nested loosening, only one that crosses spellings. + const r = resolverOver([ + { dir: '/root/real', class: 'ignore' }, + { dir: `/root/real/${NFC}`, class: 'full' }, + ]) + assert.equal(r.resolve(`/root/real/${NFC}/sub`).class, 'full') +}) + +test('resolve: nearest-governs is measured on the folded spelling, not on the declared one', () => { + // NFD is *longer in code units* than NFC for the same name, so a depth + // measured on the declared string can rank a decomposed ancestor above a + // composed descendant and invert nearest-governs. Five accented characters + // is enough: the outer entry is 16 code units decomposed against the inner + // entry's 14 composed, but 11 against 14 once both are folded. + const outer = '\u00e9\u00e9\u00e9\u00e9\u00e9' + const outerNfd = outer.normalize('NFD') + assert.ok(`/root/${outerNfd}`.length > `/root/${outer}/ab`.length) + const r = resolverOver([ + { dir: `/root/${outerNfd}`, class: 'ignore' }, + { dir: `/root/${outer}/ab`, class: 'full' }, + ]) + assert.equal(r.resolve(`/root/${outer}/ab/deep`).class, 'full') + // ...and the outer entry still governs everything the carve-out does not. + assert.equal(r.resolve(`/root/${outer}/other`).class, 'ignore') +}) + +test('resolve: folding never loosens, over every arrangement of a nested pair', () => { + // Exhaustive rather than illustrative: for every pair of classes and every + // assignment of spellings to the outer and inner entry, the folded verdict is + // at least as restrictive as the verdict the declared spellings alone give. + const RANK = { ignore: 2, 'local-only': 1, full: 0 } + const classes = /** @type {const} */ (['ignore', 'local-only', 'full']) + const spellings = [NFC, NFD] + for (const outerClass of classes) { + for (const innerClass of classes) { + for (const outerSpelling of spellings) { + for (const innerSpelling of spellings) { + for (const cwdSpelling of spellings) { + const entries = [ + { dir: `/root/${outerSpelling}`, class: outerClass }, + { dir: `/root/${innerSpelling}/inner`, class: innerClass }, + ] + const cwd = `/root/${cwdSpelling}/inner/deep` + const folded = resolverOver(entries).resolve(cwd).class + // The pre-fold answer, computed here from the same fixture by the + // plain string rule the matcher used before this change. + const preFold = declaredOnlyVerdict(entries, cwd) + assert.ok( + RANK[folded] >= RANK[preFold], + `folded ${folded} is looser than pre-fold ${preFold} for outer=${outerClass} inner=${innerClass}` + ) + } + } + } + } + } +}) + +/** + * The pre-fold rule, reimplemented from `master`: longest matching declared + * `dir` wins, ties broken by the more restrictive class, nothing matching means + * `full`. + * + * @param {readonly { dir: string, class: 'ignore' | 'local-only' | 'full' }[]} entries + * @param {string} cwd + * @returns {'ignore' | 'local-only' | 'full'} + */ +function declaredOnlyVerdict(entries, cwd) { + const RANK = { ignore: 2, 'local-only': 1, full: 0 } + const matches = entries.filter((e) => cwd === e.dir || cwd.startsWith(e.dir + '/')) + if (matches.length === 0) return 'full' + return matches.reduce((best, e) => { + if (e.dir.length > best.dir.length) return e + if (e.dir.length === best.dir.length && RANK[e.class] > RANK[best.class]) return e + return best + }).class +} + +// --- the case half, over an injected volume verdict ----------------------- + +test('resolve: case is NOT folded by default, because this volume is case-sensitive', () => { + // The correctness bug in the other direction: on Linux and on a + // case-sensitive APFS volume `Proj` and `proj` are genuinely two + // directories, and folding them would over-restrict. + const r = resolverOver([{ dir: '/root/Proj', class: 'ignore' }]) + assert.equal(r.resolve('/root/proj').class, 'full') +}) + +test('resolve: case IS folded when the volume verdict says the volume is case-insensitive', () => { + const r = resolverOver([{ dir: '/root/Proj', class: 'local-only' }], { caseInsensitiveVolume: () => true }) + assert.equal(r.resolve('/root/proj/sub').class, 'local-only') +}) + +test('resolve: a case-insensitive volume verdict still does not let a carve-out loosen a broader entry', () => { + const r = resolverOver( + [ + { dir: '/root/real', class: 'ignore' }, + { dir: '/root/real/Proj', class: 'full' }, + ], + { caseInsensitiveVolume: () => true } + ) + assert.equal(r.resolve('/root/real/proj/sub').class, 'ignore') +}) + +test('resolve: the case verdict is asked per entry, so a per-volume answer applies per entry', () => { + /** @type {string[]} */ + const asked = [] + const r = resolverOver( + [ + { dir: '/insensitive/A', class: 'ignore' }, + { dir: '/sensitive/B', class: 'ignore' }, + ], + { + caseInsensitiveVolume: (dir) => { + asked.push(dir) + return dir.startsWith('/insensitive/') + }, + } + ) + assert.equal(r.resolve('/insensitive/a').class, 'ignore') + assert.equal(r.resolve('/sensitive/b').class, 'full') + assert.deepEqual(asked, ['/insensitive/A', '/sensitive/B']) +}) + +// --- the probe itself ------------------------------------------------------ + +test('createVolumeCaseProbe is inert off darwin: constant false, and it issues no syscall', () => { + // The only claim about the probe this host can witness. Its darwin behaviour + // is not exercised anywhere in this suite and is not verified by it. + let statCalls = 0 + const probe = createVolumeCaseProbe({ + platform: 'linux', + statSync: () => { + statCalls += 1 + return { dev: 1, ino: 1 } + }, + }) + assert.equal(probe('/root/Proj'), false) + assert.equal(probe('/anything'), false) + assert.equal(statCalls, 0) +}) + +test('createVolumeCaseProbe memoizes a definite verdict per volume, not per path', () => { + /** @type {string[]} */ + const statted = [] + const probe = createVolumeCaseProbe({ + platform: 'darwin', + statSync: (p) => { + statted.push(p) + // One volume (dev 7) where every spelling names the same inode. + return { dev: 7, ino: 42 } + }, + }) + assert.equal(probe('/vol/Proj'), true) + assert.equal(probe('/vol/Other'), true) + assert.equal(probe('/vol/deep/Nested'), true) + // Two stats for the first path (the path and its case-flipped spelling), then + // one per later path to learn its `dev`, and no second probe of the volume. + assert.deepEqual(statted, ['/vol/Proj', '/vol/pROJ', '/vol/Other', '/vol/deep/Nested']) +}) + +test('createVolumeCaseProbe does not memoize an undetermined answer as the volume verdict', () => { + // A directory whose name has no cased character (`/vol/123`) admits no probe, + // but that says nothing about the volume. Caching the fallback would let one + // such path decide the verdict for every other path on the same disk. + const probe = createVolumeCaseProbe({ + platform: 'darwin', + statSync: () => ({ dev: 7, ino: 42 }), + }) + assert.equal(probe('/vol/123'), false) + assert.equal(probe('/vol/Proj'), true) +}) + +test('createVolumeCaseProbe does not memoize a non-ENOENT failure of the flipped stat', () => { + // The other undetermined branch, and the one that actually bites on a real + // volume: the directory itself stats fine, but the case-flipped spelling + // fails with something that is *not* `ENOENT` (`EACCES` on a directory the + // daemon may not traverse, `EIO` on a flaky mount). `ENOENT` would be + // informative, since it means the flipped spelling does not resolve and the + // volume is therefore case-sensitive. Any other errno says nothing at all, so + // caching it would let one transient error disable case folding for every + // path on that disk for the life of the resolver, silently reopening the gap + // this module exists to close. + let flippedFails = true + const probe = createVolumeCaseProbe({ + platform: 'darwin', + statSync: (p) => { + if (p === '/vol/Proj' || p === '/vol/Other') return { dev: 7, ino: 42 } + if (flippedFails) { + const err = /** @type {Error & { code: string }} */ (new Error('EACCES')) + err.code = 'EACCES' + throw err + } + return { dev: 7, ino: 42 } + }, + logSkip: () => {}, + }) + assert.equal(probe('/vol/Proj'), false) + // The volume verdict must still be open, so a later probe of the same `dev` + // that *can* reach an answer is believed rather than served the stale `false`. + flippedFails = false + assert.equal(probe('/vol/Other'), true) +}) + +test('createVolumeCaseProbe reports case-sensitive when the flipped spelling does not exist', () => { + const probe = createVolumeCaseProbe({ + platform: 'darwin', + statSync: (p) => { + if (p === '/vol/Proj') return { dev: 7, ino: 42 } + const err = /** @type {Error & { code: string }} */ (new Error('ENOENT')) + err.code = 'ENOENT' + throw err + }, + }) + assert.equal(probe('/vol/Proj'), false) +}) + +test('createVolumeCaseProbe fails toward the pre-fold behaviour and logs a hashed skip', () => { + /** @type {{ name: string, fields: Record }[]} */ + const events = [] + const probe = createVolumeCaseProbe({ + platform: 'darwin', + statSync: () => { + const err = /** @type {Error & { code: string }} */ (new Error('EACCES')) + err.code = 'EACCES' + throw err + }, + logSkip: (name, fields) => events.push({ name, fields: fields ?? {} }), + }) + assert.equal(probe('/vol/Secret'), false) + assert.equal(events.length, 1) + assert.equal(events[0].name, 'usage_policy.case_probe_skipped') + assert.equal(events[0].fields.error_kind, PATH_CASE_PROBE_ERROR_KIND) + assert.equal(events[0].fields.status, 'skipped') + assert.equal(events[0].fields.errno, 'eacces') + assert.match(String(events[0].fields.path_hash), /^[0-9a-f]{16}$/) + // The raw path never appears in any attribute value. + for (const value of Object.values(events[0].fields)) { + assert.ok(!String(value).includes('Secret'), `raw path leaked in ${String(value)}`) + } +}) + +// --- the structured signal on the hot path -------------------------------- + +test('resolve emits a hashed usage_policy.fold_tightened only when folding changed the verdict', () => { + /** @type {{ name: string, fields: Record }[]} */ + const events = [] + const r = resolverOver([{ dir: `/root/${NFC}`, class: 'local-only' }], { + logEvent: (name, fields) => events.push({ name, fields: fields ?? {} }), + }) + + // Same spelling: nothing to report. + assert.equal(r.resolve(`/root/${NFC}/a`).class, 'local-only') + assert.equal(events.length, 0) + + // Divergent spelling: the fold is what produced the restriction. + assert.equal(r.resolve(`/root/${NFD}/a`).class, 'local-only') + assert.equal(events.length, 1) + assert.equal(events[0].name, 'usage_policy.fold_tightened') + assert.equal(events[0].fields.hyp_operation, 'match_list') + assert.equal(events[0].fields.declared_class, 'none') + assert.equal(events[0].fields.folded_class, 'local-only') + assert.match(String(events[0].fields.cwd_hash), /^[0-9a-f]{16}$/) + for (const value of Object.values(events[0].fields)) { + assert.ok(!String(value).includes('caf'), `raw path leaked in ${String(value)}`) + } + + // An unrelated cwd nothing governs: still nothing to report. + events.length = 0 + assert.equal(r.resolve('/elsewhere').class, 'full') + assert.equal(events.length, 0) +}) + +// --- the hot path stays memoized on the lexical key ----------------------- + +test('resolve: folding happens on the cache miss only, so a repeated cwd re-reads nothing', () => { + let listReads = 0 + let caseVerdicts = 0 + const files = { [LIST]: JSON.stringify({ version: 2, entries: [{ dir: `/root/${NFC}`, class: 'ignore' }] }) } + const r = createUsagePolicyResolver({ + existsSync: (p) => Object.prototype.hasOwnProperty.call(files, p), + readFileSync: (p) => { + listReads += 1 + return /** @type {Record} */ (files)[p] + }, + localOnlyListPath: LIST, + caseInsensitiveVolume: () => { + caseVerdicts += 1 + return false + }, + now: () => 1000, + }) + for (let i = 0; i < 50; i += 1) assert.equal(r.resolve(`/root/${NFD}/a`).class, 'ignore') + assert.equal(listReads, 1) + assert.equal(caseVerdicts, 1) +})