release: gen-worker 0.42.0 (gw#615 serving-safety fix + gw#611/612/614 + wan family)

CHANGELOG.md

@@ -1,11 +1,90 @@
 # Changelog
 
+## 0.42.0 (2026-07-21)
+
+- **gw#615: disk telemetry can no longer freeze the event loop (0.40.7
+  post-seal_publish hang).** `_state_delta()` now reads only ModelStore's
+  cached `disk_usage_report()`; the actual statvfs/ref-index measurement
+  runs as a fire-and-forget `asyncio.to_thread` refresh gated to the
+  report TTL. A stalled provider volume mount leaves telemetry stale
+  instead of blocking StateDeltas, the th#965 heartbeat, and serving —
+  the 0.40.7 LTX boots that sealed+published then never served.
+- **th#767: `gen_worker.families.wan` — WanDefaults registered under
+  `wan22`** (wan-2.2 slot migration surface for inference-endpoints).
+- **gw#614: synthesized media-modality warmup coverage — multi-lane family
+  cells mint complete.** gw#612's publish gate left any endpoint whose
+  input-routed sibling lane needs media (qwen edit: an input image) unable
+  to ever publish its family cell — the declared/synthesized warmup fills
+  only required payload fields, the edit lane records calls=0, publish is
+  withheld, and every second boot re-mints (~24 min). The synthesized
+  warmup now runs a coverage pass: when a compile-target object is still
+  unexercised after the planned jobs, media VARIANTS of the same base
+  payloads (base = declared warmup payload when present, else the
+  synthesized default; exactly ONE optional image/audio field filled with
+  a generated asset, nothing else drifts) exercise the remaining lanes.
+  Driven by payload schema + compile-object coverage, no endpoint-name
+  switch; applies to mint (union cell publishes) and adopt (the sibling
+  lane proves against the cell instead of arming unproven). New
+  `warmup.media_variants`; real-inductor fresh-subprocess proof that a
+  two-lane union cell serves BOTH lanes as FX hits
+  (tests/test_cell_portability_gw611.py).
+- **gw#614: on_hello_ack model-set-diff cancel (th#961 defense in
+  depth).** Every HelloAck used to cancel + restart the residency-
+  reconcile task, killing any in-flight self_mint_compile at phase=load
+  (th#961: 4,602 cancels in 19 min). The worker now diffs the ack's
+  semantic model set (resolutions + disk_refs + snapshots + hot) against
+  the running reconcile's target: identical set → keep the task, apply
+  non-model deltas only; changed set → cancel as before.
+- **gw#612: multi-lane self-mint — publish gated on full capture coverage;
+  post-proof activity phase.** ie#501 run 26's "post-seal_publish hang" is
+  DISPROVEN on evidence: the qwen 2-lane minting worker completed setup,
+  advertised readiness (`newly_available=[generate]` hub-side 20:00:50),
+  and idled; the 2.5h wedge was hub-side — the singular compile fence saw
+  the record's TWO same-identity self-attested targets (t2i + edit riding
+  one family cell) as ambiguous and starved dispatch forever (tensorhub
+  lockstep fix: same-identity siblings collapse to one deterministic
+  pick). Worker-side real defect fixed here: the shared capture packs
+  only the graphs the warmup compiled, so a mandatory sibling lane the
+  warmup never exercised (qwen edit — no warmup modality) left the
+  published "family cell" lane-1-only, bricking every adopting boot at
+  the gw#607 per-object proof (gw#611 qwen variant, hits=1/misses=1 →
+  compile_cell_failed → release broken). `finalize_self_mint` now only
+  packs; the executor decides after the whole proof pass:
+  `publish_self_mint` when every capture-sharing object proved into the
+  cell, `withhold_self_mint_publish` (typed, loud:
+  `SELF_MINT_PUBLISH_WITHHELD`) when any sharer went unexercised — the
+  boot still serves compiled locally and re-mints next boot instead of
+  poisoning the store. New `finalize` activity phase covers the
+  post-proof tail (sibling resolution, publish decision, bookkeeping
+  through readiness) so completed activities stop reporting a stale
+  `seal_publish`.
+- **gw#611: adopt-proof counter blindness fixed; portability contract
+  pinned.** Measured (torch 2.13): with the AOT autograd cache in BUNDLED
+  mode an AOT hit serves the compiled artifact with the fxgraph counters
+  fully silent — a healthy serving adopt read `cache_hits=0,
+  cache_misses=0` to the warmup proof and fail-closed BRICKED the release
+  (th#954 SDXL second boot). `inductor_counters` now reports
+  `aot_cache_hit`/`aot_cache_miss` and the guard wrapper credits AOT-layer
+  hits as serving evidence (production pins the AOT layer off per gw#608,
+  so these stay 0 unless a config regression re-enables it — which now
+  degrades to a proven boot instead of a bricked release). The fail-closed
+  detail gains `calls=` so an orphaned/never-invoked wrapper (calls=0) is
+  distinguishable from counter-blind serving (calls>0) on the wire. New
+  real-codepath repro (tests/test_cell_portability_gw611.py): mint ->
+  pack -> fresh-process adopt -> warmup counts >=1 FX hit (CPU inductor,
+  real subprocesses), plus the bundled-AOT 0/0 mechanism pin. Hub lockstep
+  (tensorhub chaos 9e7dca8e): cells that fail their own adopt proof are
+  QUARANTINED (unattachable; next boot self-mints) instead of bricking
+  the release via model_load_failure_streak.
+
 ## 0.41.0 (2026-07-21)
 
 - **gw#613/th#965: universal app-level heartbeat (liveness layer 2).**
   ie#501 run 26 proved transport keepalive validates the gRPC library's
-  threads, not the application: a hung worker answered HTTP/2 pings for
-  2.5h while `generate` never left loading. The worker now declares
+  threads, not the application: a worker answered HTTP/2 pings through
+  2.5h of app-level silence, indistinguishable from a hung one (it was in
+  fact healthy-idle with no idle beat, starved by a hub fence bug — the
+  beat makes that diagnosis instant). The worker now declares
   `Hello.heartbeat_interval_ms=10000` and force-re-sends the full
   StateDelta every 10s from the asyncio event loop (the pgw#610
   disk-report task, promoted to the beat — never a detached thread), in

docs/compile-cache.md

@@ -9,7 +9,7 @@ split:
   compiles the declared shape set, and publishes the captured
   `TORCHINDUCTOR_CACHE_DIR` + `TRITON_CACHE_DIR` as ONE deterministic
   `.tar.gz` flavor `#inductor-<sku>-torch<maj.min>` of the family system repo
-  `_system/family-<family>`.
+  `root/family-<family>`.
 - **Consumer** — an endpoint opts in with
   `@endpoint(compile=Compile(family="flux2-klein-4b", shapes=((768,768),(1024,1024))))`.
   At load the worker seeds a VERIFIED artifact (exact-match on family, SKU,
@@ -29,8 +29,9 @@ cold compilation through an explicit library argument. There is no serving
 environment fallback that can bypass scheduler attachment or W8A8 fencing.
 
 Trust: compiled artifacts are CODE. Only platform jobs may publish to
-`_system/*` (invoke-time destination-write preflight + cap-token repo+owner
-gate + the tenant slug grammar bars `_system`). Tenant custom-code endpoints
+`root/*` (invoke-time destination-write preflight + cap-token repo+owner
+gate + `root` is a platform-reserved slug tenants cannot claim). Tenant
+custom-code endpoints
 get per-release private caches (same-principal rule) — not implemented yet.
 
 Family keying: caches key on the traced graph + shapes, not weights — one

examples/marco-polo/src/marco_polo/main.py

@@ -49,6 +49,18 @@ async def marco_polo_slow(
 
         raise ValidationError(f"expected 'marco', got {data.text!r}")
 
+    async def marco_polo_wedge(
+        self, ctx: RequestContext, data: MarcoPoloInput
+    ) -> MarcoPoloOutput:
+        """th#965 layer-2 liveness probe: an ASYNC handler that BLOCKS the
+        event loop with a synchronous sleep — the control loop that owes
+        heartbeats goes silent while the gRPC transport threads keep
+        answering keepalive pings. A th#965 hub must fire
+        worker_heartbeat_lost within its miss window and recycle the pod;
+        this handler never returns inside any sane enforcement window."""
+        time.sleep(1800)
+        return MarcoPoloOutput(response="unreachable")
+
     async def marco_polo_stream(
         self, ctx: RequestContext, data: MarcoPoloInput
     ) -> AsyncIterator[MarcoPoloOutput]:

proto/CONTRACT.md

@@ -160,7 +160,11 @@ that worker's assigned requests (attempt+1) unless the worker returns and
 re-claims them via `Hello.in_flight` reconcile first. A pod-backed worker
 that never reconnects within the stall window is enforced as
 `worker_disappeared`. Keepalive proves ONLY the gRPC library's threads:
-ie#501 run 26's hung worker answered HTTP/2 pings for 2.5h.
+ie#501 run 26's worker answered HTTP/2 pings through 2.5h of app-level
+silence — a silent worker is indistinguishable from a hung one at the
+transport (run 26's turned out to be healthy-idle with no idle beat,
+starved by a hub fence bug; the beat makes the two cases distinguishable
+in seconds instead of hours).
 
 **Layer 2 — universal app-level heartbeat (hung process).**
 
@@ -193,9 +197,10 @@ Heartbeats arriving but ANY function still in `loading_functions` with NO
 open activity (no running ActivityUpdate kind, no active model download) for
 `DefaultActivityStallAfter` (10min) → `worker_activity_stalled` with reason
 `loading_no_open_activity`, same enforcement. Completing an activity without
-starting the next one or declaring readiness is silence, not health. This
-closes the run-26 blind spot: activity-coverage gaps (gw#612 class) degrade
-the stall reason, never the detection.
+starting the next one or declaring readiness is silence, not health.
+Activity-coverage gaps (gw#612 class) degrade the stall reason, never the
+detection — a worker genuinely stuck between its last activity and
+readiness can no longer ride unbounded.
 
 **Load balancer requirements.** Any proxy/LB between W and O's gRPC port MUST:
 - forward HTTP/2 PING frames end-to-end (no PING termination at the proxy —
@@ -586,7 +591,7 @@ A missing/empty operation ID or digest fails closed as
 before any download, cache seeding, pipeline wrapping, or resident-state mutation.
 
 **ADOPT_COMPILE_CACHE** (hot adoption, #567): `ref` is a compile-cache flavor
-ref — `_system/family-<f>#inductor-<sku>-torch<maj.min>`. W downloads the
+ref — `root/family-<f>#inductor-<sku>-torch<maj.min>`. W downloads the
 artifact snapshot, verifies its key (family/SKU/torch/triton/libs/producer
 gen-worker version + low-VRAM prep mode, gw#391 — the prep flags are traced
 into the graphs) against its own runtime and resident pipelines, seeds the
@@ -674,7 +679,7 @@ re-baselines. A positive `residency_generation` plus non-empty
 manufactures observed identity from desired config or a mutable tag target.
 `ADOPTED` is also not a residency tier: it reports one-shot success of ADOPT_COMPILE_CACHE for
 a compile-cache ref (whose bytes independently report DOWNLOADING/ON_DISK
-like any snapshot download). O MUST NOT feed `_system/family-*` compile-cache
+like any snapshot download). O MUST NOT feed `root/family-*` compile-cache
 refs into ordinary model-failure availability handling. An optional lane may
 stay eager; mandatory W8A8 remains unavailable until exact compiled evidence
 changes and must never be marked function-ready from the failed cell.
@@ -782,9 +787,9 @@ stale instance, and only then promotes the newer disk identity.
 **Compile-cache snapshots (#569).** When a release's endpoint declares
 `compile=Compile(family=...)` and boot-attach is enabled (opt-in; default OFF
 — boot-time attach worsens TTFI, hot adoption is the primary path), O MAY add
-the resolved `_system/family-<f>#inductor-<sku>-torch<maj.min>` snapshot to
+the resolved `root/family-<f>#inductor-<sku>-torch<maj.min>` snapshot to
 `RunJob.snapshots` keyed by that ref, alongside the model snapshots. W
-recognizes the key by the `_system/family-<f>#inductor-` prefix for the
+recognizes the key by the `root/family-<f>#inductor-` prefix for the
 declared family, downloads it like any snapshot, and seeds it before pipeline
 load; verification failure ⇒ eager, never an error. The same ref may arrive
 via `ModelOp{ADOPT_COMPILE_CACHE}` (hot adoption, §4).

proto/worker_scheduler.proto

@@ -555,7 +555,7 @@ enum ModelOpKind {
   reserved 1, 2, 3;
   reserved "MODEL_OP_KIND_DOWNLOAD", "MODEL_OP_KIND_LOAD", "MODEL_OP_KIND_UNLOAD";
   // Hot-adopt a torch.compile cache artifact: ref is a compile-cache flavor
-  // ref (`_system/family-<f>#inductor-<sku>-torch<maj.min>`). The worker
+  // ref (`root/family-<f>#inductor-<sku>-torch<maj.min>`). The worker
   // downloads+verifies+seeds the cache, re-wraps the already-resident modules
   // of endpoints declaring compile family <f>, runs one warmup trace, and
   // answers ModelEvent{ADOPTED, duration_ms}. ANY failure => stay eager and

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "gen-worker"
-version = "0.41.0"
+version = "0.42.0"
 description = "A library used to build custom functions in Cozy Creator's serverless function platform."
 readme = "README.md"
 license = "MIT"

src/gen_worker/activity.py

@@ -35,6 +35,10 @@
 PHASE_INDUCTOR_COMPILE = "inductor_compile"
 PHASE_WARMUP_FORWARD = "warmup_forward"
 PHASE_SEAL_PUBLISH = "seal_publish"
+# gw#612: post-proof tail — sibling-lane resolution, publish decision,
+# residency/target bookkeeping through readiness. Phases are logged
+# verbatim hub-side (only kinds are enumerated in worker_activity.go).
+PHASE_FINALIZE = "finalize"
 
 # Default watchdog cadence; the hub's stall rule (~10 min) tolerates many
 # missed beats.

src/gen_worker/compile_cache.py

@@ -17,7 +17,7 @@
 Artifacts are FAMILY-keyed (settled 2026-07-06): torch.compile caches key on
 the traced graph + shapes, not the weights, so one artifact serves every
 fine-tune of a model family. They live in a system-owned repo per family
-(``_system/family-<family>``), one flavor per (SKU, torch) cell — and they
+(``root/family-<family>``), one flavor per (SKU, torch) cell — and they
 are CODE: only the platform's first-party compile job publishes shared ones.
 
 Artifact = deterministic ``.tar.gz``::
@@ -234,12 +234,12 @@ def system_repo(family: str) -> str:
     fam = str(family or "").strip()
     if not fam:
         raise ValueError("compile-cache family must be non-empty")
-    return f"_system/family-{fam}"
+    return f"root/family-{fam}"
 
 
 def parse_cell_ref(ref: str) -> Tuple[str, str]:
     """(family, flavor) from a system cell ref
-    (``_system/family-<f>[:tag][@digest][#<flavor>]``) via the ONE ref
+    (``root/family-<f>[:tag][@digest][#<flavor>]``) via the ONE ref
     grammar (gw#492); ('', '') when the ref is not a system-family ref."""
     from .models.refs import parse_model_ref
 
@@ -248,7 +248,7 @@ def parse_cell_ref(ref: str) -> Tuple[str, str]:
     except ValueError:
         return "", ""
     th = parsed.tensorhub
-    if th is None or th.owner != "_system" or not th.repo.startswith("family-"):
+    if th is None or th.owner != "root" or not th.repo.startswith("family-"):
         return "", ""
     return th.repo[len("family-"):], th.flavor or ""
 
@@ -556,17 +556,32 @@ def _reset_inductor_latch() -> None:
 
 
 def inductor_counters() -> Dict[str, int]:
-    """This process's inductor FX-graph cache counters (monotonic). The delta
+    """This process's compiled-artifact cache counters (monotonic). The delta
     across a warmup is the honest adopted-vs-silently-eager signal (gw#391):
-    zero hits means the seeded cell never served the trace."""
+    zero hits means the seeded cell never served the trace.
+
+    gw#611: the AOT-autograd layer is a SECOND serving surface. In bundled
+    mode an AOT hit loads the compiled artifact without ever consulting
+    FxGraphCache (measured on torch 2.13: fxgraph counters fully silent on a
+    served call), so a proof reading only fxgraph_* sees hits=0/misses=0 on
+    a healthy serving cell and fail-closes it — the th#954 SDXL second-boot
+    release-bricking shape. AOT hits are therefore reported alongside
+    (``aot_cache_hit``/``aot_cache_miss``) and count as serving evidence.
+    Production pins the AOT layer OFF (gw#608 portability), so these stay 0
+    unless a config regression re-enables it — in which case a served
+    warmup must still prove, never brick."""
     try:
         from torch._dynamo.utils import counters
 
         c = counters["inductor"]
-        return {
+        out = {
             k: int(c.get(k, 0))
             for k in ("fxgraph_cache_hit", "fxgraph_cache_miss", "fxgraph_cache_bypass")
         }
+        a = counters["aot_autograd"]
+        out["aot_cache_hit"] = int(a.get("autograd_cache_hit", 0))
+        out["aot_cache_miss"] = int(a.get("autograd_cache_miss", 0))
+        return out
     except Exception:
         return {}
 
@@ -1185,8 +1200,12 @@ def record_success(before: Optional[Dict[str, int]]) -> None:
         stats = counters_delta(before, inductor_counters()) if before is not None else {}
         with lock:
             signal["successful_calls"] = int(signal.get("successful_calls", 0)) + 1
+            # gw#611: an AOT-layer hit serves the artifact without an
+            # FxGraphCache lookup (bundled mode: fxgraph counters silent);
+            # it is serving evidence, never a disproof.
             signal["cache_hits"] = int(signal.get("cache_hits", 0)) + max(
-                0, int(stats.get("fxgraph_cache_hit", 0)))
+                0, int(stats.get("fxgraph_cache_hit", 0))) + max(
+                0, int(stats.get("aot_cache_hit", 0)))
             signal["cache_misses"] = int(signal.get("cache_misses", 0)) + max(
                 0, int(stats.get("fxgraph_cache_miss", 0)))
 
@@ -1267,8 +1286,12 @@ def record_success(before: Optional[Dict[str, int]]) -> None:
         stats = counters_delta(before, inductor_counters()) if before is not None else {}
         with lock:
             signal["successful_calls"] = int(signal.get("successful_calls", 0)) + 1
+            # gw#611: an AOT-layer hit serves the artifact without an
+            # FxGraphCache lookup (bundled mode: fxgraph counters silent);
+            # it is serving evidence, never a disproof.
             signal["cache_hits"] = int(signal.get("cache_hits", 0)) + max(
-                0, int(stats.get("fxgraph_cache_hit", 0)))
+                0, int(stats.get("fxgraph_cache_hit", 0))) + max(
+                0, int(stats.get("aot_cache_hit", 0)))
             signal["cache_misses"] = int(signal.get("cache_misses", 0)) + max(
                 0, int(stats.get("fxgraph_cache_miss", 0)))