Usage-policy gate folds path spellings a volume treats as one directory (NFC verified, case unverified)

llp/0049-hypignore-usage-policy.spec.md

@@ -68,6 +68,16 @@ matched over the set of spellings that denote it (as-given plus
 symlink-resolved), and the most restrictive verdict any spelling produces wins,
 so a `.hypignore` cannot be escaped by reaching its subtree through a symlink.
 
+**Extended-by:
+[LLP 0050 §normalization](./0050-ignore-enforced-in-adapters.decision.md#normalization)**:
+symlinks are not the only way a filesystem spells one directory several ways.
+The ancestor walk is unaffected by the rest (it `stat`s each candidate rather
+than comparing two strings), but the machine-local list's membership test
+compares a `cwd` a client reported against a `dir` a CLI declared, so each of
+the spellings above is compared in a **folded** form (Unicode-normalized, and
+case-folded on a volume probed case-insensitive). Like canonicalization, the
+fold can only ever make the verdict more restrictive.
+
 ## Classes {#classes}
 
 | Class | V1 | Meaning |

llp/0050-ignore-enforced-in-adapters.decision.md

@@ -194,6 +194,13 @@ governs this directory?" is `scopeGoverns`, which has to be the same predicate
 `resolve` used, or `policy show` names a governor the gate did not use and
 `policy unset` refuses to remove an entry the gate is enforcing.
 
+Symlinks are not the only way a filesystem spells one directory several ways.
+[§normalization](#normalization) widens the same set again, by Unicode
+normalization and per-volume case, through the same two-pass guard: the two are
+one mechanism in the code, not two stacked ones. Read that section before
+touching `selectGoverning`, and note that the fold it adds stops at the gate
+while the canonicalization described here does not.
+
 ## Why not the gateway
 
 - The gateway is the **provider-agnostic** proxy ([LLP 0016](./0016-ai-gateway.decision.md)).
@@ -212,6 +219,222 @@ Two copies of a privacy-critical matcher drift apart. A single core module with
 one test suite is the safer home; sibling-to-sibling plugin imports would be
 worse coupling than both importing core.
 
+## The set of spellings that denote a directory is volume-dependent {#normalization}
+
+The shared matcher compares **strings**. A filesystem hands one directory
+several strings, and which mechanisms apply is a property of the **volume**,
+not of the path and not of the platform:
+
+| mechanism | folded by | volume-dependent? |
+|---|---|---|
+| symlinked components | `realpath(2)` | no |
+| Unicode normalization (NFC vs NFD) | nothing in `node:fs` | yes, and folding is **unsafe** where it does not apply, but harmless *at the gate* (see below) |
+| case | nothing in `node:fs` | yes, and folding is **unsafe** where it does not apply |
+
+`realpath(2)` resolves symlinks and does nothing else. On a default macOS
+(APFS) volume the kernel accepts, `stat`s, and `chdir`s to spellings it will not
+fold: `Proj` and `proj` are one directory, and `Café` spelled NFC (U+00E9) and
+NFD (`e` + U+0301) are one directory. So the gate could be handed a `cwd` whose
+spelling differs from the spelling a machine-local entry was declared with and
+return `full` for a directory the user opted out of. That is not an exotic
+case: macOS frameworks and Finder-derived paths emit NFD while typed and
+JSON-transported paths are usually NFC, and the two paths this gate compares are
+produced by **different processes at different times** (a CLI resolving a mark,
+versus a client reporting a `cwd`).
+
+**Decision: list membership compares a folded spelling of both sides.** The fold
+is `src/core/usage-policy/fold.js`:
+
+1. **NFC unconditionally, and only because this is the gate.** It is a total
+   function of the string, needs no filesystem access, cannot fail, and is the
+   identity on a path that is already composed. What makes it safe here is
+   **not** that NFC and NFD always name one directory. They do not: on every
+   Linux volume this codebase targets, `caf` + U+00E9 and `cafe` + U+0301 are
+   two directories with two inodes, and both can exist in one parent
+   (demonstrated on an ext4-backed overlay host: distinct `ino`, distinct
+   contents, both listed by `readdir`). Folding them together therefore *can*
+   merge two genuinely different directories, exactly as unconditional case
+   folding would.
+
+   It is safe at the gate anyway, for a reason specific to the gate: the
+   resolved class is `max(declared, folded)` (`selectGoverning`, and the argmax
+   discussion below), so a fold that merges two distinct directories can only
+   ever **over-restrict**, i.e. decline to record a directory that was in fact
+   permitted. That is a usability cost
+   and never a privacy or data-loss one. Case is put behind a probe rather than
+   given the same treatment because case aliasing is far more likely to collide
+   with a real, deliberately-distinct sibling (`Makefile` vs `makefile`) than a
+   normalization difference is, not because NFC folding is universally sound.
+
+   **Do not reuse `foldPath` in a predicate where widening is not free.** In a
+   *deletion* predicate (`hyp purge`) or a *disclosure* predicate, widening
+   removes or reveals rows for a directory the user did not name, and the
+   `max()` argument above does not apply. Closing the purge seam below needs
+   either a per-volume normalization-insensitivity probe (no such probe exists;
+   the current one answers only the case question) or a darwin-only guard.
+   See "Not covered".
+2. **Case only behind a per-volume probe.** Case-sensitivity is a property of
+   the mounted volume: an APFS volume can be formatted case-sensitive and every
+   ext4 volume is. Folding it unconditionally would merge two genuinely
+   different directories, which is a correctness bug in the other direction. The
+   probe compares the `dev`/`ino` of a path against a case-flipped spelling of
+   its last segment, memoizes by `dev`, and is inert (constant `false`, no
+   syscall) off darwin. An undetermined probe resolves to "case-sensitive",
+   which is the pre-fold behaviour, so a failed probe can only fail to *add*
+   reach.
+
+The fold must **distribute over the path separator**, since its only consumer is
+a segment-aware prefix test: `fold(a + '/' + b) === fold(a) + '/' + fold(b)`.
+Both halves do (`/` is a starter that participates in no canonical composition,
+and `toLowerCase` maps it to itself), and the property is asserted rather than
+assumed.
+
+### A widened spelling must only ever add restriction
+
+Producing a folded spelling is necessary but **not sufficient**. The
+machine-local list's nearest-governs step is an **argmax over match depth**, and
+an argmax discards verdicts instead of merging them. A less restrictive entry
+that gains reach through its folded spelling can become the deepest match and
+displace a broader restrictive entry that already governed: a `--sync` carve-out
+spelled NFC would punch a hole in a private tree spelled NFD, and the directory
+would **start recording and forwarding**. Nothing about "compare folded
+spellings" prevents that on its own.
+
+So nearest-governs is evaluated **twice**, once over the spellings exactly as
+declared (which reproduces the pre-fold verdict) and once folded, and the **more
+restrictive of the two answers wins**, the declared one breaking a class tie
+because it is the spelling the user typed. The resolved class is therefore
+`max(pre_fold, folded)` on the restrictiveness lattice by construction, which
+makes "folding never opens the gate" structural rather than a property someone
+has to remember
+([LLP 0049 §fail-safe](./0049-hypignore-usage-policy.spec.md#fail-safe)).
+
+The visible cost is that a **nested loosening does not cross spellings**: a
+carve-out has to be declared in the same spelling as the entry it carves out of.
+`hyp policy show` reports the class actually in force, so it is diagnosable.
+
+Specificity is measured on the **folded** spelling, not the declared one. NFD is
+longer in code units than NFC for the same name, so a declared-string depth can
+rank a decomposed ancestor above a composed descendant and invert
+nearest-governs.
+
+### Cost
+
+The per-`cwd` memo is keyed on the **lexical** path and consulted **before** any
+folding, so a cache hit costs exactly what it did before. Entry spellings and
+the per-volume case verdict are computed once per list parse, inside the TTL
+window LLP 0049 R6 already bounds.
+
+`String.prototype.normalize('NFC')` is roughly 60 ns on a pure-ASCII path, but
+that is the case where the fold does nothing, so it is the wrong number to plan
+against. When the string is **already** composed, normalize verifies and
+returns: about 60 ns for a short ASCII path and about 90 ns for a 114-character
+path with accented segments. When it is genuinely decomposed, which is exactly
+the macOS case this section exists for, it has to recompose: about **550 ns**
+for a 134-character NFD path, roughly 9x the ASCII figure, and it scales with
+length (about 9.6 us for a pathological 1000-character all-decomposed path).
+
+That is affordable only because of where it sits. Folding is on the cache-**miss**
+path (once per `cwd` per TTL window) and on the list parse (once per entry per
+window), never per exchange. A miss over a 20-entry list with a long NFD `cwd`
+measures about 8.2 us before this change and 9.7 us after. A caller that ever
+moves the fold onto a per-row or per-exchange path has to re-measure with a
+decomposed path, not an ASCII one.
+
+### Relationship to the symlink class
+
+This is the same shape [§canonicalization](#canonicalization) arrives at for
+symlinks, for the same reason, and the two were found by the same review. They
+are independent in *what* they fold - `realpath` cannot fold case or
+normalization, and folding cannot resolve a symlink - but not in *how* they are
+guarded, so they landed as one mechanism rather than two.
+
+The composition is a `map`, not a second set: `realpath` yields a **set** of
+spellings, the fold is a **function** on a spelling, so the widened set is the
+fold's image of the canonical set (`listScope` in `matcher.js`). And there is
+exactly **one** two-pass argmax guard, evaluated over declared-and-unfolded
+versus widened-and-folded, because the displacement hazard is identical whichever
+mechanism gave a carve-out its extra reach. Running the guard twice would buy
+nothing; running it once over the composed set is what makes
+`class = max(pre_widening, widened)` hold for both mechanisms at once.
+
+### Not covered
+
+The fold is applied at the **gate** (`resolve` / list membership) and nowhere
+else. The one-shot CLI membership sites (`hyp ignore --check`, `policy show`,
+`policy unset`) and `hyp purge --subtree` now route through the shared
+spelling-aware predicates [§canonicalization](#canonicalization) introduced
+(`scopeGoverns`, `sameDirectory`, `governingListEntry`), so they fold **symlinks**
+with the gate, but those predicates deliberately stop short of `foldPath`
+(`canonicalScope` in `matcher.js`, as against `listScope`). On a case-insensitive
+or NFD-carrying volume they can therefore still disagree with the gate about
+normalization and case.
+
+That is the "widening is not free" rule above, not an oversight: the single
+shared predicate is now the right *place* to add the fold, and adding it is still
+gated on a per-volume normalization-insensitivity probe (or a darwin-only guard)
+that does not exist yet. The gap below is stated in terms of that.
+
+The disagreement is bounded in one direction and not in the other, and the
+difference matters enough to name each site:
+
+- **The CLI can never promise more protection than the gate delivers.** The
+  lexical predicate matches a subset of what the folded one does, and the gate's
+  class is `max(declared, folded)`, so any entry the CLI finds the gate also
+  found. There is no spelling on which `--check` reports a directory protected
+  while the gate forwards it.
+- **`hyp ignore --check` / `policy show` report the right class and can name the
+  wrong scope.** The class comes from `resolve()`, so it is folded and correct.
+  Only the "which listed directory governs this?" lookup (`governingListEntry`)
+  is unfolded, so when the entry reaches `cwd` only by folding it falls back to
+  the queried path. The class is right; the governing directory shown, and the
+  residual row count scoped to it, are narrower than the truth. Note that the
+  row count makes this a *disclosure* predicate, which is why it does not simply
+  inherit the gate's fold either.
+- **`policy unset` / `unignore --local-only` can refuse to remove an entry the
+  gate is enforcing**, when the user spells the path the other way. It reports
+  "not governed" and exits 0. That fails toward privacy: the opt-out stays on.
+- **`hyp purge <path>` (the subtree target) silently retains rows it was asked
+  to delete**, when the rows were recorded under a different spelling of the
+  target. This is the one site that fails **away** from privacy: the user asked
+  for data to be deleted, the command reports success, and the rows remain.
+  Observed end to end (rows recorded NFD, purge argument NFC, and the reverse;
+  also a case alias):
+
+  ```
+  # the argument is typed NFC; the rows were recorded under the NFD spelling.
+  # the two render identically, which is the whole problem.
+  $ hyp purge ~/café/proj --yes
+  purged 0 rows from 0 partitions
+  $ echo $?
+  0
+  ```
+
+  Nothing is written to stderr and the exit status is 0, so the outcome is
+  indistinguishable from "that directory had nothing cached". Note the
+  inversion: a purge that *succeeds* prints the resurrection warning on stderr,
+  so the failing case is the **quieter** of the two. It is unchanged from the
+  pre-fold behaviour rather than introduced here, and it is tracked separately;
+  it is the reason this seam should not stay open for long.
+
+- **`hyp purge --ignored` is already covered by this change**, because that
+  target classifies each row through `resolver.resolve(row.cwd)` rather than
+  through a lexical prefix test, so it inherits the fold. Verified against
+  `master`: with an `ignore` entry declared NFC and rows recorded NFD, `master`
+  purges 0 rows and leaves the row, and this branch purges it. So marking the
+  directory and running `hyp purge --ignored` is the durable workaround for the
+  subtree gap above until that gap is closed.
+
+Whoever closes the subtree gap should note that it is **not** a matter of
+dropping `foldPath` into the predicate. Purge deletes, so widening the match is
+not free the way it is at the gate (see "NFC unconditionally, and only because
+this is the gate"): on a Linux volume, folding would delete cached rows for a
+genuinely different sibling directory that differs only by normalization. The
+fix needs the fold gated on the volume actually being normalization-insensitive.
+`scopeGoverns` - which already reroutes this purge call site for the symlink
+class - is the right place to put it, and `canonicalScope` is the one line that
+has to change once such a probe exists.
+
 ## Consequences
 
 - Code that lands this carries `@ref LLP 0050 [implements]` on the adapter

src/core/usage-policy/canonical.js

@@ -1,32 +1,23 @@
 // @ts-check
 
-import { createHash } from 'node:crypto'
 import nodeFs from 'node:fs'
 import path from 'node:path'
 
 import { Attr } from '../observability/attrs.js'
 import { getLogger } from '../observability/logger.js'
 
+// One path digest for the whole usage-policy seam, so a hash in a
+// `canonicalize_failed` line and a hash in a `case_probe_skipped` line name the
+// same path when they name the same path.
+import { hashPath } from './fold.js'
+
 /**
  * `error_kind` for a `realpath(2)` that could not fully canonicalize a path.
  * Never fatal: the caller keeps the lexical spelling and the gate stays at
  * least as restrictive as it was (see {@link canonicalSpellings}).
  */
 export const PATH_CANONICALIZE_ERROR_KIND = 'path_canonicalize_failed'
 
-/**
- * Short one-way digest of a path, so a canonicalization failure is diagnosable
- * (which path, how often, which errno) without dev telemetry ever carrying a
- * raw local path. Same discipline as the `usage_policy.export_drop` aggregate
- * in `src/core/cache/storage.js`.
- *
- * @param {string} p
- * @returns {string}
- */
-function hashPath(p) {
-  return createHash('sha256').update(p).digest('hex').slice(0, 16)
-}
-
 /**
  * The `errno` code of a filesystem error, as a lowercase token suitable for a
  * log attribute (`enoent`, `eacces`, `eloop`), or `unknown`.