Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions plans/tecnix-target-eval-caching/explainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -341,7 +341,9 @@ The cache is a single SQLite database with one physical row family:
DependencyShards(gitDir, resolver, argsKey, shard → multi-target history blob)
```

Target discovery (§9) is stored in the same rows, under a reserved key whose candidates carry the discovered target list as a payload; discovery thereby shares the lookup, validation, history, and compaction machinery of ordinary targets rather than maintaining a parallel implementation. Ordinary target candidates carry a payload too: the `drvPath` their evaluation produced. A proven candidate whose drv is still a valid store path therefore answers `tecnixTargets` value queries without evaluating — the value is rebuilt from the drv itself, exactly as `import <drvPath>` would build it — while a candidate whose drv is missing from the store is simply an ordinary miss. The key contains no commit. The `argsKey` column holds the canonical JSON encoding of the caller's `args` value; this is sound as a key because the resolver receives that same value, so results can depend on the arguments only through content that is, by construction, the key.[^ambient-inputs] Validity across trees is established entirely by the closure-matching procedure of §4.1.
Target discovery (§9) is stored in the same rows, under a reserved key whose candidates carry the discovered target list as a payload; discovery thereby shares the lookup, validation, history, and compaction machinery of ordinary targets rather than maintaining a parallel implementation. The key contains no commit. The `argsKey` column holds the canonical JSON encoding of the caller's `args` value; this is sound as a key because the resolver receives that same value, so results can depend on the arguments only through content that is, by construction, the key.[^ambient-inputs] Validity across trees is established entirely by the closure-matching procedure of §4.1.

Ordinary target candidates carry a versioned `{drvPath, outputName}` payload. Both fields are forced under source tracking, so a source read that changes only output selection still invalidates the candidate. The payload consists of `TXTV1`, a NUL byte, the recipe path, another NUL, and the selected output name. Its fields are viewed directly in the stored bytes, without parsing a JSON object on each hit. A target-value hit also requires a locally valid recipe and an output of that name. Importing the recipe and selecting the recorded output preserves its Nix string contexts — the dependency metadata attached to strings. An unsupported payload, missing recipe, or invalid output selection is an ordinary miss.

[^ambient-inputs]: Ambient inputs that a pure evaluation can still observe — `builtins.nixVersion`, the store directory — are deliberately *not* part of the cache key. This aligns with Nix's existing flake evaluation cache, whose key is likewise content-only. Changes to the evaluator itself, or to Tecnix semantics, are instead handled by bumping the version in the cache's filename (`tecnix-eval-cache-v2.sqlite`), which orphans old rows wholesale rather than mixing results from two evaluator versions in one database.

Expand All @@ -351,7 +353,7 @@ Sharding is a row-size compromise. With `N` targets, `S` shards, and an average

The first implementation deliberately uses flat newest-first candidate history rather than a decision trie. The common case is that the latest candidate still matches, and the history bound is small. In that case, a trie adds another index to build, validate, and explain without reducing the expensive part of validation: computing the current fingerprint once per unique path. The per-run fingerprint memo already makes repeated path checks cheap. A trie over `(pathId, fingerprintId)` predicates could become worthwhile if measurements show many stale candidates per hot target and repeated pair scans dominate warm lookup, but it is not needed for the initial sharded blob design.

Dependency and discovery blobs begin with the magic bytes `TXDC` (for "TecniX Dependency Closure"). The magic serves as the format's self-identifier: foreign, corrupted, or out-of-date blobs are rejected immediately, and rejection is a cache miss rather than an error. The blob bytes are laid out so they are already the data structure used by validation:
Dependency and discovery blobs begin with the magic bytes `TXDC` (for "TecniX Dependency Closure") followed by a version field. The magic serves as the format's self-identifier: foreign, corrupted, or out-of-date blobs are rejected immediately, and rejection is a cache miss rather than an error. The current blob version is 2; version 1 omitted the selected-output input from target closures and could silently serve the wrong output of a multi-output recipe. The blob bytes are laid out so they are already the data structure used by validation:

```
header: TXDC magic, fixed format marker, counts, section offsets
Expand Down
12 changes: 7 additions & 5 deletions plans/tecnix-target-eval-caching/guardrails.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,10 +92,10 @@ Use this as a review checklist for source-dependency tracking and target-eval ca
- **Dependency-only queries must not force target values.**
- Continue to support `includeDependencies = true; includeTargets = false;`.

- **A target-value cache hit needs the closure proof plus the drv.**
- A proven candidate may carry the evaluated target's `drvPath` as its payload.
- Reusing it requires that same complete closure proof plus the drv still being a valid store path; when target values are requested, anything else (no payload, malformed payload, garbage-collected drv) is an ordinary miss and re-evaluates.
- Cached target values are derivation-shaped (the `import <drvPath>` surface: `type`, `name`, `drvPath`, `outPath`, `outputs`, per-output attrs); resolver attributes outside that contract are not preserved on a value hit and must not be relied on.
- **A target-value cache hit needs the closure proof, recipe, and selected output.**
- Store a versioned `{drvPath, outputName}` payload, with both fields forced under source tracking.
- Reusing it requires the complete closure proof, a locally valid recipe, and an output of the recorded name. Unsupported payloads, absent recipes, and invalid selections are ordinary misses.
- Cached values are the selected output of the imported recipe, with its Nix string contexts intact. Other resolver attributes are not preserved on a value hit and must not be relied on.

- **Keep legacy `unsafeTectonixInternal*` compatibility isolated from the new Tecnix cache/history design.**

Expand All @@ -104,4 +104,6 @@ Use this as a review checklist for source-dependency tracking and target-eval ca
- **Do not add migrations for unshipped development cache formats.**
- During development, incompatible local rows should miss or be wiped and rebuilt.

- **Keep the current development blob marker fixed unless the cache format becomes a shipped compatibility contract.**
- **Bump the blob version when old observations become unsafe, not just when the format changes.**
- The version field is a compatibility boundary: old blobs that omit an input the current evaluator needs must be rejected, not silently reused.
- A version bump orphans all local rows; during development that is safe because the database is disposable.
4 changes: 2 additions & 2 deletions plans/tecnix-target-eval-caching/walkthrough.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ sequenceDiagram

> **Structure: the per-run fingerprint memo.** A thread-local table from path to current fingerprint. It exists because the closures of many targets, and the historical candidates for one target, share most of their paths; each unique path is fingerprinted once per run — a git object-identifier and file-mode read, which is itself inexpensive — and every subsequent occurrence is a hash lookup. Validation cost therefore scales with the number of *unique* paths candidate scanning asks about, not with the total number of historical closure entries. (In a worldtree sandbox the clean tree is the daemon's immutable FUSE projection rather than a git repository: directory fingerprints come from a tree-oid xattr, and regular-file fingerprints from a blob-oid xattr when the daemon serves one, falling back to hashing content as a git blob, memoized in memory for the run — identical fingerprint strings, identical validation; see the explainer's §7.)

If one candidate fully matches, the journey ends here: that candidate identifies the matching historical closure, the output is built directly from its pair stream, and everything described in the remaining sections is skipped — the resolver, every force, every frame, all interning. This asymmetry accounts for the difference of several orders of magnitude between warm and cold runs. The `absent` entries participate in the search as well: a path that the target once probed and did not find is checked to still be absent, so a newly created file fails the proof in exactly the way an edited one does.
If one candidate fully matches, a dependency-only query builds its answer directly from that candidate's pair stream. A target-value query also checks the versioned `{drvPath, outputName}` payload, requires a locally valid recipe, and selects the recorded output from its imported value. This preserves the output path and Nix string context; an unsupported payload or unavailable output makes the query a miss instead. A usable hit skips the resolver and the source-tracking work below. The `absent` entries participate in validation too: a newly created file invalidates a candidate that previously observed its absence.

For the purposes of this walkthrough, suppose no complete candidate matches. The lookup is a miss, and evaluation must run — under observation.

Expand All @@ -100,7 +100,7 @@ Two preparations precede the target itself.

**The dirty overlay is established.** A single `git status` invocation partitions the tree: clean paths will be served from the git object store at the pinned commit, and modified paths from disk. If `git status` fails, evaluation fails. Assuming a clean tree in that situation would allow stale rows to validate against a tree that does not reflect reality, so the failure is made visible instead.

The resolver is then applied to `"//services/api"`, the resulting value's `drvPath` is forced, and control descends into the evaluator.
The resolver is then applied to `"//services/api"`. Its `drvPath` and selected `outputName` are forced while the tracking context remains active. The output name can depend on a source read that does not affect the recipe path, so tracking cannot stop after forcing `drvPath`.

## 6. Step ④: Inside Evaluation

Expand Down
4 changes: 2 additions & 2 deletions src/libexpr/include/nix/expr/tecnix/eval-cache.hh
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ struct TecnixDependencyUpsert
std::string_view target;
const DependencyClosure * dependencies;
/** Stored as the candidate's payload (discovery's target-name JSON, or a
target's evaluated drvPath); empty for none. */
target's evaluated drvPath/outputName result); empty for none. */
std::string payload;
};

Expand All @@ -92,7 +92,7 @@ public:
Value * toValue(EvalState & state) const;

/** The matched candidate's payload (discovery's target-name JSON, or a
target's evaluated drvPath), if any. */
target's evaluated drvPath/outputName result), if any. */
std::optional<std::string_view> payload() const;

private:
Expand Down
Loading
Loading