release 0.47.0: th#1031 cell_selection_bug self-mint recovery by PaulFidika · Pull Request #372 · cozy-creator/python-gen-worker

PaulFidika · 2026-07-23T07:50:14Z

Version bump + changelog for th#1031 (PR #371, merged 1ddf3a2). Mirrors the th#883 two-PR release recipe (fix PR, then a release PR to keep the fix diff reviewable).

…master) (#386) * gw#587/gw#605: pin — a fresh Executor (no setup yet) advertises pre-load candidate cell lookups Unit-level the candidates work (records are eagerly created at construction); the LIVE gap (4 consecutive boots never received the stored cell — every boot re-minted) is therefore in the candidate->StateDelta->release-config-> attach chain, not in cell_lookups() itself. Forensics + eliminated theories in tracker gw#605. * compile-cache: platform org _system -> root (lockstep with tensorhub th#940 chaos 0e4f6b1e) system_repo now derives root/family-<f>; ref parsing, self-mint publisher owner, docs, proto contract, and test fixtures move in lockstep. Only the org segment changes — repo/flavor grammar is byte-identical. * gw#608: disable the AOT autograd cache in the compile path — cell portability requires the FX cache as the lookup surface; pack asserts its FX entries before publish ROOT CAUSE (cracked dry, three same-key captures from three pods): the AOTAutogradCache key hashes fx_kwargs[get_decomp_fn] via the function REPR — which embeds the process memory address (ASLR), so AOT keys can NEVER match across processes. inductor/fxgraph entries were bit-identical across all pods (8/8); inductor/aotautograd keys had zero overlap. On the consumer, the AOT-layer miss recompiles without consulting the portable on-disk FX entries: cache_hits=0, cache_misses=8 on two independent hosts against a byte-good cell (fail-closed held — never eager). Fix: _disable_aot_autograd_cache() (env TORCHINDUCTOR_AUTOGRAD_CACHE=0 + torch._functorch.config.enable_autograd_cache=False) at capture_env/ seed_env AND apply() — symmetric for producer capture and consumer seeding. Lookups collapse to FxGraphCache (proven portable); cells shrink (~50% was AOT payload); fresh processes pay a cheap AOT re-analysis, not an inductor recompile. Hardening (coordinator ruling): a minting boot proves its EXECUTION — finish_fleet_mint now also proves its ARTIFACT: refuse to publish a capture with zero fxgraph entries, and (when the proof supplies its compiled-graph count via finalize_self_mint(expected_graphs=pipe_misses)) a partial capture. Revert-turns-red tests for both. (5 pre-existing chaos test failures reproduced on untouched HEAD.) * gw#608: adopt-test fake warmup writes the fxgraph store the artifact assertion requires (one entry per miss) — the 0.40.4 CI failure was this un-updated fake, not the capture path * th#960/pgw#609 Phase 2: tests/harness/ + P1-P7 (stream lifecycle, residency reconcile, slot binding precedence, CAS multiwriter, discovery lock, cancel/ backpressure, boot smoke) Harness extracted per design: hub_double.py generalizes FakeScheduler from test_worker_grpc_e2e.py (the only double, at the true hub gRPC boundary); blob_host.py is a real blake3-verified HTTP blob server; toy_endpoints.py consolidates toy fixtures (echo/stream/GPU-slot-yield/Hub-bound/Slot-bound/ billable); subprocess_runner.py generalizes the gw#591 boot-smoke pattern. P3 absorbs PR #353 (pgw#606/th#938) and PR #307 (pgw#583) as real hub-double dispatches (no mocking of gen_worker internals, unlike the original unit tests) - revert-verified red against lifecycle.py's pre-fix state, restored clean. P1/P2/P4/P5/P6/P7 absorb assertions from test_worker_grpc_e2e.py, test_shared_cas_root_multiwriter.py, and test_boot_smoke_gw591.py. Found + fixed in the harness itself: WorkerHarness must set the TENSORHUB_CACHE_DIR env var (gen_worker.models.cache_paths reads the process-wide get_settings() singleton, not the per-worker Settings instance) - passing tensorhub_cache_dir= to load_settings() alone silently no-ops, and every hub-double test defaults to a fresh per-call temp dir so downloaded bytes never leak between tests. 40 passed, 1 skipped (pgw#605 heartbeat: proto fields not generated yet) in ~33s local. Existing test files not touched or deleted (Phase 3). * th#960/pgw#609 Phase 2: P8 (convert publish contract) + P9 (result upload metrics), plus ruff cleanup + a hub_double(file_base_url=) knob P9 needs P8 (tests/convert/test_p8_convert_publish_contract.py, hermetic fake_hub): dtype-mismatch cast honesty (pgw#589/th#901, condensed from test_publish_as_is_dtype.py), classifier corpus table (pgw#593), and an xfail(strict=True) test-first pin for pgw#566 (open bug: normalize_adapter _state_dict only passes unet_config to converters with a NAMED parameter - diffusers' real SDXL lora_state_dict takes **kwargs only, so the SGM block remap silently never runs). pgw#569's W8A8 verifier ULP gate is a documented skip - no factored-out comparator to test hermetically without a real w8a8 artifact round trip. P9 (tests/test_p9_result_upload_metrics.py): inline vs blob_ref by size alone over a real hub-double + real local media-upload dedup sink (no S3 part scripting, matching test_media_upload_owner.py's pattern); JobMetrics typed usage survives both wire forms. Needed hub_double(file_base_url=) so the sink URL rides the FIRST HelloAck instead of a racy post-connect mutate. ruff clean (unused imports, one E731 lambda->named helper). mypy: src/gen_worker clean (untouched); new tests/harness + test files carry the same untyped- fixture-param convention as the rest of tests/ (mypy only gates src/gen_worker per CI, tests/ was never in scope). Full new suite (P1-P9 + P8's own file): 50 passed, 1 skipped (pgw#605, proto fields not generated), 1 xfailed (pgw#566, fix open) in ~31s local. * th#960/pgw#609 Phase 3: delete 4 test files fully superseded by P3/P4/P8 test_slot_boot_precedence_th938.py, test_model_slot_identity_gate.py -> superseded by test_p3_slot_binding_precedence.py (same assertions, over a real hub-double boundary instead of monkeypatched Executor.store internals). test_shared_cas_root_multiwriter.py -> superseded by test_p4_cas_multiwriter_integrity.py (same 3 tests reproduced verbatim + 1 new row). tests/convert/test_publish_as_is_dtype.py -> superseded by tests/convert/test_p8_convert_publish_contract.py (same 3 core tests against the same fake_hub). Deletions only, no other changes. Full deletion manifest (all 166 files audited, reasoning for what's NOT deleted) is in the th#960/pgw#609 tracker checkpoint - the bulk of the pre-existing suite tests production surface (CLI, LoRA runtime, quantization lanes, VRAM/RAM admission, media/upload, Slot/discovery SDK, tests/convert producer suite) that no P-test covers, so it stays pending explicit direction on the larger absorb-or-accept-loss effort that would be needed to delete it safely. * pgw#610: proto — measured per-tier DiskUsageReport on StateDelta (lockstep with th#962) * pgw#610: measured per-tier disk telemetry in every StateDelta (th#962 lockstep) - statvfs on the real mounts (CAS root = container tier; ismount-guarded fill source volume when attached; NFS tier reserved) — never a declared size - safely-reclaimable = ref-index bytes of DISK-tier refs inactive AND not in the desired set (the disk-GC LRU's eligible set); reuses ref-index bytes, no tree rescans - capacity_generation bumps only on a measured (64 MiB-quantized) shape change; rides Hello.state and every StateDelta; 30s periodic refresher is edge-suppressed * pgw#609/th#960 Phase 4: rewire CI to one suite, one lane Manual dispatch + PR-into-master only (drafts/doc-only paths run nothing); drop the on-push trigger. Same gates (mypy gating, ruff non-gating per debt note, HTTP-timeout guard, llama-server, uv build); pytest targets the whole tests/ dir so P-suite additions land automatically. * Revert "pgw#609/th#960 Phase 4: rewire CI to one suite, one lane" This reverts commit c39efb69b82d967393bd3f3243ce328d954380f7. * th#960/pgw#609 Phase 2b: absorb ~26 incident-pinned rows into P1/P2/P3/P5/ P6/P8/P9 + 2 new consolidated files, ahead of the bounded aggressive deletion sweep Bucket-level triage per the coordinator's ruling (Paul's directive: minimal greenfield set, ~92% removal). Absorbed only distinct bug-classes carrying a real pgw#/gw# incident number or a revert-catches-it story; untagged "tested thoroughly once" coverage goes without absorption (git history is the archive). P1 (+6): auth-rejection exit/retry-within-window, permanent-precondition fast-exit, hello-ack-deadline reconnect, not-leader redirect, hub-restart reconnect - absorbed from test_worker_grpc_e2e.py (#372) ahead of its deletion. P2 (+3): setup-failure FnUnavailable+recovery, host-RAM-failure wire ordering (th#807) from test_worker_grpc_e2e.py; corrupt-load quarantine+ retry-once (gw#408, J17 flood) from test_snapshot_corruption.py. P3 (+1): bare-ref prefetch binding resolution (#377) from test_executor_prefetch_binding.py. P5 (+1): heavy-dep stub contract (pgw#506) from test_discovery_heavy_deps.py. P6 (+1): SendQueue capacity-generation fencing from test_worker_grpc_e2e.py. P8 (+5): diffusers root-allinone skip (e2e J7 ENOSPC), standalone-component canonical-weight selection (gw#426), transformers sharded-index onnx/pickle exclusion, gguf explicit-quant refusal, non-safetensors-format refusal family - from test_classifier.py ahead of its deletion. P9 (+1): token-bound-owner routing (J19 run34) from test_media_upload_owner.py. New consolidated files (coordinator-authorized, one per lane with no P-test home): tests/test_quantization_lanes.py (W8A8/W4A4/FP8 contract + gw#409 promote-device-integrity, from test_w8a8.py/test_w4a4.py/ test_fp8_and_emergency_loading.py/test_promote_device_integrity.py) and tests/convert/test_convert_producer_contract.py (clone concurrency gw#442, disk preflight gw#462, download-skip bank th#592, publish resilience gw#462, from the tests/convert/ producer suite). Also reverted a premature local-only Phase 4 CI commit (c39efb6, from a prior session before an interruption) - never pushed, out of THIS issue's authorized scope ("do not touch CI yet"); preserved in history via revert, not discarded. Full new-suite gates: mypy clean, ruff clean, each new/changed file green in isolation. Deletion commit (the ~125 files these rows supersede) follows as a separate, deletions-only commit per the coordinator's mechanics. * th#960/pgw#609 Phase 3b: aggressive deletion sweep — 125 files (32,618 LoC) superseded or absorbed per the coordinator's ruling Every file here is either fully superseded by test_quantization_lanes.py / test_convert_producer_contract.py, absorbed as rows into P1/P2/P3/P5/P6/P8/ P9 (chaos b8274a1), or carries no incident pin (git history is the archive, per the coordinator's bound). Full manifest, bucket-by-bucket reasoning, and the dangling-import sweep are in the th#960/pgw#609 tracker Checkpoint 6, pushed before this commit. Deletions only, no other changes. Foreign-WIP clusters untouched: compile_cache/executor_adopt/fleet_cells (this issue's own claim-checkpoint note) and test_disk_telemetry_pgw610.py (landed on chaos by another lane during this session). Final state: 14 test files (3,293 LoC) + tests/harness/ (6 files) + conftest.py x2 + fake_hub.py, target was ~12-15 files per the coordinator. * pgw#610: lint — typed enum cast + drop unused import * pgw#610: CONTRACT.md — document StateDelta.disk_usage / DiskUsageReport * pgw#610: guard drain's disk-report cancel with getattr (stubbed Lifecycles skip __init__, same convention as _cancel_residency_reconcile) * gw#608 residual: AOT-cache disable must be process-global — torch 2.13 config user overrides are thread-local ContextVars Live disproof (0.40.5, B200, 2026-07-21): mint capture still packed 8 ASLR-keyed aotautograd entries and the store-served sibling still failed (warmups=1, cache_hits=0, cache_misses=8) — the 0.40.4 disable ran on the arming thread; the warmup compile thread saw the default True. The env var is read once at torch import (env_name_force), so post-import it is inert. Fix: entrypoint sets TORCHINDUCTOR_AUTOGRAD_CACHE=0 before any torch import (binds all threads + compile-worker subprocesses); _disable_aot_autograd_cache additionally mutates the installed config entry's env_value_force (process- global) for already-imported-torch embedders. Cross-thread revert-turns-red test added (fails on the 0.40.5 shape, passes with the fix). Dry evidence: fxgraph keys 8/8 identical across ALL five banked packs (0.40.1 x3 / 0.40.3 / 0.40.5); aotautograd level-2 keys 0/8 everywhere. samples/selfmint-proof/gw0405-run/. * th#964: family lane policy twin — bare conv-UNet (sd1/sd2/sdxl) bindings AUTO-pick scale-free #fp8 locally gen_worker.models.ladder gains family_root (modelfamily.Root twin) and the conv-UNet w8a8 exclusion table; a maybe_rebind_family_fp8 fold (hub-parity AUTO pick over the resolve's sibling_flavors, sm_89+ only, bf16 stays the sub-floor default; pins untouched) runs in the local CLI resolve ahead of the GGUF fold over one shared resolve. WorkerResolvedRepo carries the resolve's model_family (slot-declared family as fallback); select_gguf no longer lets an AUTO-ineligible w8a8 row suppress the GGUF pick. Shared vectors: none in this repo; no format change. * gw#613/th#965: universal app-level heartbeat — layer 2 of the 3-layer liveness contract (0.41.0) Hello declares heartbeat_interval_ms=30000; the pgw#610 disk-report task is promoted to the beat: it force-re-sends the full StateDelta every 30s from the asyncio event loop (the control loop that owes progress, never a detached thread), starts at startup() entry so boot hangs are covered, and keeps beating through drain until the stream closes. Edge-triggered deltas still flow as before; the beat is the same message re-sent unchanged. Contract §3 rewritten to the 3-layer model (lockstep tensorhub c03a531d). Tests: hello cadence, force-send bypasses edge suppression, beats flow while a startup coroutine is parked (the gw#612 shape), drain keeps beating until close. * gw#613 retune: 10s beat x 6 misses; disk stats measured every 30s riding every beat HEARTBEAT_INTERVAL_MS 30000 -> 10000 (hub reaps at 6 misses, ~60s). The pgw#610 statvfs/ref-index scan keeps its 30s measurement cadence behind a TTL cache so the 10s beat never turns it into a hot loop; the report still rides every StateDelta. Contract §3 event-loop discipline clause added — already true in 0.41.0: setup/warmup/compile/residency/GC all run off-loop (executor._to_thread_complete / asyncio.to_thread). * gw#612: gate self-mint publish on full capture coverage; finalize phase; run-26 root cause corrected (0.42.0) Run 26's filed post-seal_publish hang is disproven: the worker completed setup and advertised readiness; the wedge was the hub's singular compile fence treating the 2-lane record's same-identity self-attested targets as ambiguous (tensorhub lockstep fix). Real worker defect fixed: a shared mint capture missing an unexercised mandatory sibling's graphs was still published as the family cell, bricking every adopting boot (gw#611 qwen hits=1/misses=1). finalize_self_mint packs only; the executor publishes iff every capture sharer proved in, else withholds typed+loud. New 'finalize' activity phase covers the post-proof tail to readiness. * gw#613: soften run-26 narrative — app-silent, not proven-hung (gw#612 root cause: healthy-idle, hub fence starvation) * gw#613: marco_polo_wedge — event-loop-blocking liveness probe for th#965 layer-2 drills * gw#611: credit AOT-layer hits in the adopt warmup proof; calls= in the fail-closed detail; cross-process portability repro Measured mechanism for th#954's release-bricking 0/0: bundled AOT cache serves a fresh-process adopt with fxgraph counters fully silent, so a HEALTHY serving cell read cache_hits=0/cache_misses=0 and fail-closed. inductor_counters now surfaces aot_cache_hit/aot_cache_miss and the guard wrapper counts AOT hits as serving evidence (AOT stays pinned off per gw#608; a config regression now degrades to a proven boot, not a brick). Fail-closed detail carries calls= to split orphaned-wrapper (calls=0) from counter-blind serving (calls>0). New tests: real mint->pack->fresh- subprocess adopt asserts >=1 FX hit; bundled-AOT 0/0 mechanism pin; guard credits AOT hits (red on revert). Hub lockstep: tensorhub chaos 9e7dca8e quarantines cells failing their own adopt proof. * th#767: gen_worker.families.wan — WanDefaults (animegen lane) Second family after SdxlDefaults: steps/guidance/guidance_2/max_guidance for the Wan 2.2 A14B MoE envelope. Registered under tensorhub's canonical architecture root "wan22" (not a Compile(family=) shape-grid string) so repo-metadata PUT validates against the matching tensorhub schema (companion wan22.schema.json on tensorhub chaos). Lets the wan-2.2 endpoint's T2V function migrate to the Slot model and give AnimeGen-T2V its own distilled recipe (8 steps, guidance=1.0, guidance_2=1.0) via repo metadata instead of branching on checkpoint identity. * gw#614: synthesized media-modality warmup coverage (union family cells publish) + on_hello_ack model-set-diff cancel (A) _run_synthesized_warmup coverage pass: compile-target objects still unexercised after the planned jobs get media VARIANTS of the same base payloads (warmup.media_variants: one optional image/audio field filled with a generated asset, nothing else drifts) — input-routed sibling lanes (qwen edit) mint into the shared capture, so the gw#612 gate publishes the union cell instead of withholding forever; adopts prove the sibling lane instead of arming unproven. (B) th#961 defense in depth: a HelloAck whose semantic model set (resolutions+disk_refs+snapshots+hot) matches the running reconcile's target no longer cancels it (in-flight self_mint_compile survives benign plan rewrites); changed sets cancel as before. * gw#615: fix event-loop-blocking disk telemetry — the 0.40.7 LTX post-seal_publish hang pgw#610/th#962's disk_usage_report() ran statvfs()/stat() synchronously inline in Lifecycle._state_delta(), which many hot paths call directly with no event-loop offload — including right after a self-mint's seal_publish. The provider-attached VOLUME fill-source is a network-backed mount that can stall for minutes under load (exactly what a self-mint's weight download + cell pack produce). A stalled statvfs() there freezes the whole event loop: every StateDelta, RunJob dispatch, and even the th#965 heartbeat (same loop) stops until the syscall returns. ModelStore.disk_usage_report() now only reads a cache; the actual measurement (_measure_disk_usage_report) runs exclusively via refresh_disk_usage_report(), off-loop through asyncio.to_thread. Lifecycle kicks that refresh as a fire-and-forget background task (never awaited inline), gated to the existing ~30s TTL, so a stalled mount delays only its own telemetry freshness — never any send, dispatch, or heartbeat. Root-caused via git diff v0.40.5..v0.40.7 (executor.py/lifecycle.py) plus gw#612's hub-log forensics for the qwen case: gw#612 itself was hub-side (dispatch-fence starve, corrected root cause, no worker hang) and its worker-side companion fix addresses a different defect (poisoned family-cell publish on the NEXT boot). This is a distinct, genuinely-new gen-worker regression introduced by a70e7c9 (wave-2 telemetry), reproduced locally with no GPU via a stalled-statvfs monkeypatch. Tests: tests/test_disk_telemetry_pgw610.py (new test_disk_usage_report_never_measures_only_reads_the_cache, test_state_delta_never_blocks_event_loop_on_a_stalled_mount; existing tests updated to the cache/refresh split) + tests/test_heartbeat_gw613.py (new test_beats_flow_while_disk_measurement_is_stalled; existing TTL test updated to the fire-and-forget refresh). All revert-turns-red verified. Full suite 458 passed/5 skipped; ruff+mypy clean. * gw#608: FX-key forensics — a store-served proof failure names the diverging FxGraphHashDetails component; compile_seconds + per-object proof counts in the fail-closed detail * gw#608: clamp fx-divergence detail under the 2000-char activity error cap * gw#608: exhaustive fx_cache_failure_report — counts + same-key re-save sibling diff + load probes discriminate B1/B2 in one boot * gw#608 FIX: self-mint arm is transactional over the process-global cache env — no-target siblings decline before any dir mutation, arm failure restores the prior env, and a delivered-cell-seeded process never opens a capture (the store-served 8/8-miss root cause) * gw#585: consume tensorhub v4 private-input manifests — v3->v4 hard cut (PR #320 port onto chaos) Proto adopted byte-for-byte from tensorhub chaos 07e5520b (protocol 4, RunJob.input_assets=15 — 14 taken by th#913 lane — InputAsset fields 1-6; CLI NDJSON stays v1); pb2 regenerated with grpcio-tools 1.82.1. Worker resolves the exact ordered manifest via one bounded resolve call under the attempt-scoped capability, verifies size/BLAKE3/MIME/kind/decode, preserves opaque refs, sets only local_path, attempt-owned temp cleanup; endpoint build rejects Asset-bearing set/frozenset and non-string maps; deterministic traversal (declaration order, list index, sorted string map keys). Full local suite 558 passed; mypy/ruff/ timeout-lint clean. uv.lock: mechanical 0.42.0 catch-up to pyproject. * gw#585: marco-polo-attach — private-input echo probe (blake3/size/local_path, optional in-flight delay) for the v4 e2e lane * gw#585: GEN_WORKER_INTERNAL_OBJECT_HOSTS — exact-host allowlist for resolver-minted private-input URLs (datacenter-internal object stores); caller public transports keep the full SSRF gate * gw#585: marco-polo-attach reads via os.fspath (Asset is os.PathLike; no .path() method) * gw#608: fix publish-event race in the self-mint adopt test fake — publish() fills the published dict before signaling the event (CI KeyError: 'bytes') * pgw#617: hierarchical slot bindings — load base composition, substitute components from ModelBinding.components - proto: ModelBinding.components (field 5, map component -> canonical ref; mirrors tensorhub th#980); pb regenerated - ModelRef.component_overrides (tensorhub-only, sorted pairs) — part of the binding identity, so a component-only rebind derives a new instance/ residency identity; flat bindings byte-identical - executor: _effective_spec stamps overrides per dispatch; setup materializes each override ref from its own snapshot and injects the loaded module via the gw#479 components= from_pretrained mechanism; pins/held refs/digests/compile-cell binding facts include override refs - loading.load_component_override: class from the BASE model_index.json, weights from the override tree's subtree/root - typed ComponentSubstitutionError (unknown component, non-CAS ref, non-worker-loaded slot) - registry: selected_by= slots may omit default_checkpoint (th#980 relaxation mirror; unblocks ie#524 sweep of wan-2.2/sdxl/z-image) - tests: hub-double flat->substitute->flat identity proof, typed unknown- component refusal, registry relaxation (564 green) * pgw#617: mypy — dedicated slot_comps var in _effective_spec * gw#619: report a typed hardware-unsuitable fact to the hub before the boot-time CUDA probe's silent exit entrypoint.py's boot sequence (gw#529 cuda_probe) logged GEN_WORKER_CUDA_PROBE_FAILED and exit(1)'d with zero orchestrator contact — invisible to every layer of telemetry (th#986: ~20 pod deaths whose cause, CUDA-13 images on old-driver hosts per th#988, was unattributable). On a probe failure the worker now dials the hub and sends ONE HardwareUnsuitable WorkerMessage in place of Hello (cozy.scheduler wire addition, tensorhub th#988 companion commit) before exiting — bounded best-effort (~7s worst case: 2 attempts x 3s + 1s backoff), never a reason to delay the exit materially, and a silent fallback when the hub is unreachable or predates this field (old-hub FAILED_PRECONDITION rejection). cuda_probe.py: classify_probe_failure() — typed reason_class vocabulary (torch_unavailable | cuda_unavailable | driver_too_old | cuda_error | unknown) derived from CudaProbeResult.reason; driver_too_old matches torch's own "driver too old (found version ...)" message verbatim (the exact th#591/th#979/th#988 signature). hardware_report.py (new): builds the report (nvidia-smi/NVML for driver_version+gpu_name — reads the driver directly, independent of the broken CUDA runtime being reported on; torch_version/torch_cuda_version; gen_worker_version; image_digest/instance_id from Settings; worker_id/ release_id from Settings or JWT claims) and dials it over a one-shot grpc.aio Connect stream. entrypoint.py: calls report_hardware_unsuitable() right where the probe failure was previously silently fatal; logs a cuda_probe_hardware_report startup phase either way. Tests: tests/test_hardware_report.py (classify_probe_failure vocabulary; build_hardware_report degrades safely without nvidia-smi and reads driver identity when torch.cuda is down; report_hardware_unsuitable delivers to a real hub-double socket, resolves identity from JWT claims, and returns promptly-false against both an old-hub protocol rejection and a connection-refused unreachable hub). tests/test_boot_smoke_hardware_report.py (true end-to-end: a real `python -m gen_worker.entrypoint` subprocess whose probe fails on THIS box's actual driver/torch mismatch dials a real hub socket, and the hub-unreachable fallback still exits without hanging). tests/harness/hardware_report_hub.py: shared real-socket test servicers (recording_hub, old_hub, closed_port_addr). Full suite green (578 passed) plus the extended gw#591 boot-smoke contract (tests/test_boot_smoke_gw591.py, tests/test_p7_boot_smoke.py) confirmed unaffected. * ie#522: non-inference functions must never report loading_functions() _warmup_plan already excludes non-"inference" kinds from boot warmup (spec.kind != "inference"), but available_functions()/loading_functions() didn't share that gate: a declared-but-never-dispatched conversion function (a release can bundle many; only a few get invoked) sat in loading_functions() forever, since nothing was ever going to warm it. The hub's th#965 layer-3 stall watchdog takes loading_functions() at face value and kills a worker after 10 minutes of "no open activity" on any name in that list — including a name nothing was ever loading. Reproduced live: a wan-2.2 conversion release bundling z-image-w8a8-quantization alongside clone-huggingface got its pod killed at 10m15s three times running, on three different hosts, each time mid a legitimate multi-GB clone-huggingface transfer. Fix: conversion/training/dataset-kind functions are always available (matching the existing spec.slots per-dispatch-serveability carve-out just above) and never counted as loading. * ie#522: watchdog evidence blind to I/O-bound download+load phases The activity watchdog's default liveness evidence was process CPU seconds only. Two real activities are honestly alive while burning near-zero CPU: an I/O-bound network model fill (large composites, tens of GB, CPU-light by design), and the subsequent on-disk load into GPU (safetensors mmap + device placement — third-party diffusers/torch code with no progress hook). Both read as "stalled" to the hub's th#965 layer-3 rule after 10 minutes of no heartbeat, even mid genuine progress. Reproduced live twice: identical ~9.5min self_mint_compile CancelledError at phase=load, two different RunPod hosts. Fixed three ways: - _process_cpu_evidence now sums live (not just reaped) child-process CPU via psutil — covers inductor's async_compile subprocess workers, which resource.RUSAGE_CHILDREN can't see while they're still running. - note_progress() lets an I/O callback (model-download byte ticks) heartbeat the current activity directly, independent of the CPU-sampling watchdog thread; wired into cozy_snapshot.py's _on_bytes network-byte path. - _process_io_evidence adds process disk I/O bytes (psutil io_counters) to the default combined evidence — a universal, non-invasive signal for the on-disk load phase that needs no app-level hook into third-party loading code at all. Revert-turns-red verified: reverting activity.py/cozy_snapshot.py makes exactly the 3 new evidence tests fail (zero-CPU download, zero-CPU disk load, live-child-CPU), while the pre-existing dead- activity-still-stalls test keeps the detector honest. Full suite green (584 passed, 5 skipped, 1 xfailed). * gw#621: progress registry — named monotonic counters on the 10s beat + self-diagnosis New gen_worker.progress: phases register named monotonic counters (download:<ref> bytes w/ snapshot-known totals, evidence:<kind> riding the watchdog, warmup:jobs, infer:steps, upload:bytes); family prefix maps to a per-phase self-diagnosis window (code constants). activity.on_beat rides lifecycle's 10s heartbeat: one counter-carrying ActivityUpdate per beat — frozen counters included (the hub kills on counter non-advancement) — and self_stalled=true + stalled_for_ms when even the freshest counter is past its window (kill on confession, never CPU inference). Additive ActivityUpdate fields 10-16 (counter, unit, done, total, rate, self_stalled, stalled_for_ms; th#994 carries the proto source). Counters are activity- owned (finished at activity end). Downloads now carry bytes_total on the FIRST DOWNLOADING event when the snapshot knows sizes — no more 'started total=0' then silence. Also completes ie#522's 7640551, which swept the in-flight activity.py half of this work into its commit — gen_worker.progress ships here. * gw#621: tests — registry windows/self-diagnosis, counter beats, hub-double e2e confession Revert-turns-red for the ie#522 false-positive class: a CPU-quiet byte-advancing download is visibly alive on the wire (counter beats, never self_stalled) over a REAL Worker + hub double; a frozen counter still confesses, and local behavior is unchanged (the hub owns the kill). * gw#621: progress_beat re-checks _done — a beat racing completion never re-opens a terminal activity hub-side * ie#522: arm_compile() no-ops (never raises) with no active arming scope Endpoint setup() declares intent by calling arm_compile() unconditionally (sdxl, wan-2.2, ...); whether that intent is active is the release's decision — an eager registration (no compile=Compile(...) declared) already opens ArmingScope(None, ...) as a documented no-op, but arm_compile() itself still raised RuntimeError when called with no active scope. Reproduced live: wan-2.2's eager smoke (dropped compile= to skip a 51.5min self-mint compile grid per Paul's ruling) crashed at warmup/phase=load on this exact raise. Fix: no active scope -> log at info, return False (never armed), same as every other "stays eager" path. Revert-turns-red verified (4 tests updated/added in test_arm_compile_pgw517.py). Full suite green (594 passed, 5 skipped, 1 xfailed). * pgw(th#1004): @variant_of handler annotation — variant pairing emitted into the discovery manifest (variant_of/variant), dangling/self/chain targets fail the build * ie#522: WanDefaults gains an optional shift field (scheduler flow-match shift) Live finding #3: the AnimeGen card's own 8-step recipe pins FlowMatchEulerDiscreteScheduler(shift=3.0) alongside its distillation LoRA -- a step-distilled lineage needs a non-default scheduler shift, which nothing in the wan22 inference-defaults vocabulary carried. shift: Optional[float] = None -- every non-distilled repo's metadata and the resolved-defaults fallback leave the pipeline's own scheduler config untouched; only a repo that explicitly publishes shift gets one applied at the endpoint's resolution site (wan-2.2 main.py, separate commit). Mirrors tensorhub-chaos's wan22.schema.json shift addition. * ie#522 fix: de-flake the zero-cpu watchdog test for constrained CI runners test_watchdog_survives_zero_cpu_download_via_note_progress was flaky by construction: in its 0.3s real-clock window the 5s note_progress rate-limit floor allowed exactly ONE beat, so the >=2 assertion silently depended on incidental watchdog CPU/IO evidence noise for the second beat — present on a 32-core dev box, absent on 2-core GitHub runners (confirmed by per-thread beat attribution: ticker contributes 1, the rest come from the watchdog thread; beats=1 reproduces even locally outside pytest). Deterministic rewrite, intent intact (zero-CPU download must not go silent while note_progress ticks arrive): drop the rate-limit floor for this test (the floor is covered by its own test), pin watchdog evidence to a constant zero so every counted beat is attributable to note_progress alone (airtight revert-turns-red — verified: neutering note_progress fails it with 0 beats), and wait event-driven on beat count with a 10s deadline instead of a fixed wall-clock window. Verified 20x unconstrained, 10x taskset -c 0, 5x taskset -c 0,1; full file green; mypy clean. * pgw#622: eager-while-compiling with hot-swap — novel request shapes serve immediately A novel input signature at a compiled target no longer stalls 30-60s behind Dynamo+Inductor: the guard routes it to the eager original, one background thread warms the compiled callable with a zero-filled dummy batch (separate CUDA stream, nice +10), and a successful warm atomically hot-swaps the signature to the compiled path. Each completed warm repacks the live cache root and republishes the cell under the same key so the fleet never compiles that (shape, GPU, lane) again. Sequential compile-then-serve is kept for: the boot warmup-proof window, mandatory quantized lanes (w8a8/w4a4), tight VRAM headroom (degrade, never OOM), regional mode, and signature-vocabulary explosions. * release: gen-worker 0.45.0 — pgw#622 eager-while-compiling with hot-swap * release: 0.45.0 uv.lock version sync * th#1017: SdxlScheduler gains lcm/euler_trailing distilled-regime schedulers * th#1017: @endpoint regimes= decorator surface + manifest emission Per-method {name: (regime,...)} dict on class handlers (mirrors warmup=), bare tuple on the function form, default ("standard",). Validated against {standard,v_prediction,distilled}; EndpointSpec.regimes carries the resolved per-handler tuple; discover.py emits "regimes" only when it differs from the default (absent = ["standard"]). * th#1017: ResolvedSlot.regime + executor-level regime backstop resolve_slot/resolve_slots thread inference_regime (default "standard") into ResolvedSlot.regime, sourced from the hub resolve response's inference_regime field (hub_client.WorkerResolvedRepo, mirrors model_family; "" on hubs not sending it). RegimeMismatchError raises when a resolved checkpoint's regime is outside the invoked function's declared regimes= — wired as a backstop in executor._resolve_slots_kwargs via allowed_regimes=spec.regimes. * th#1017: converter stamps regime-correct scheduler config run_clone/build_flavor_tree accept an inference_regime hint (payload field of the same name); writer.apply_regime_scheduler_config rewrites a produced diffusers flavor's scheduler/config.json: v_prediction sets prediction_type + rescale_betas_zero_snr, distilled sets trailing timestep_spacing. No-op for standard or a tree with no scheduler component. Updated the P8 build_flavor_tree test stub to accept the new kwarg. * th#1017: tests — regimes decorator surface, manifest, resolution, converter Real discovery/registry/slot/convert codepaths (no mocking): decorator validation errors, class/function manifest round-trip + default-omitted, ResolvedSlot.regime + RegimeMismatchError backstop, hub_client inference_regime parsing, and scheduler-config stamping for all three regimes. * gw#623: only cancel an in-flight residency load when the ACTIVE model leaves the set Root cause of the ~9.5min (and ~15s-burst) warmup/self_mint_compile CancelledError at phase=load on nearly every serving boot: _semantic_model_key hashed full snapshot serializations INCLUDING presigned file URLs, which the hub re-signs on every ~15s release-config rebuild and deliberately excludes from its own HelloAck semantic hash. Every resent ack carrying rotated URLs (or any sibling-ref churn, e.g. the th#1021 preposition/seed fight) looked like a model-set change and killed the in-flight load. - _snapshot_content_key: digest + (path, size, blake3) per file; URL excluded - level-triggered reconcile: passes re-read the freshest desired set; a benign mid-load update signals a re-pass instead of cancelling - cancel only when the actively-loading item left the set or changed snapshot identity / precision resolution (_active_work_still_desired) Tests (revert-turns-red verified): URL rotation keeps the load, sibling ref added mid-load keeps the load then converges, active ref leaving cancels, active-ref resolution repick cancels. * gw#624: free cancelled load/warmup attempts' memory before the retry Live: 5 cancelled load retries on one worker climbed container RAM 3%->97% and VRAM to 83.86GB (OOM on an 80GB card) — each cancelled attempt's partially loaded modules stayed alive (pinned by the CancelledError's traceback and by uncollected reference cycles) while the retry loaded a fresh copy on top. - _to_thread_complete: drop the joined Task reference on the cancel path so the discarded load result is not pinned by the traceback frame during rollback - _injection_kwargs: release shared-component holds acquired before the failure (they never reached rec.shared_keys, so vacate could not) - _rollback_failed_setup: gc.collect + cuda.empty_cache after vacate, and schedule an allocation purge - _setup_locked: run the purge (gc + empty_cache) under the load lock BEFORE the next attempt allocates — a retry provably starts from baseline Tests (revert-turns-red verified): cancelled _to_thread_complete result is unreachable while the exception lives; cancelled ensure_setup's buffers are gone at the next attempt's setup entry (gc disabled for the duration). * release: gen-worker 0.46.0 — th#1017 inference regimes * th#1031: cell_selection_bug recovers via self-mint instead of retry-blocking forever A self-requested cell whose graph signature drifts from this runtime's own (cell_key has no graph-shape axis, so structurally different sdxl graphs can share one key) used to raise CellSelectionBugError straight out of fleet_cells.enable_compiled, aborting the whole arm attempt. For a mandatory w8a8/w4a4 lane that becomes CompiledLaneUnavailableError, which fails setup entirely and gets retried from scratch on the next request/job -- hitting the identical stale delivered cell again, forever. Live pod_events proves this: 54 cell_selection_bug rows across 2 releases/9 pods over 24h+ (predating pgw#622 by ~12h, so unrelated to it), all rejecting one stale signature (c3c4395265ad) that nothing was ever able to overwrite. ArmOutcome now carries the caught CellSelectionBugError; enable_compiled falls through to self-mint (the ordinary MISS recovery) instead of raising. The th#883 invariant stays wire-visible -- callers still send the cell_selection_bug ModelEvent/pod_event -- it just no longer blocks serving. Self-mint's own fail-closed safety net (no CUDA/toolchain/target) is untouched, so a genuine mint impossibility on a mandatory lane still refuses, now chained from the selection bug via __cause__. Threaded through both compile-arming seams: the worker-loaded slot path (_enable_compiled) and the self-loaded pipeline path (ArmingScope/ arm_compile, what sdxl actually uses) via a new selection_bugs map so neither silently drops the alarm. * th#1043: joint precision fit for shared-component multi-lane loads Sequential per-lane loading let the first lane in a gw#479 shared- component group (qwen-image's t2i/edit) consume free VRAM at native precision, starving the second lane into an offload placement the shared-component invariant refuses outright ("shared-component lanes require resident placement"). Decide fp8 storage for the whole group up front, against its combined footprint, before any lane loads. * th#1043: strict_vram refuses a load-time CUDA OOM instead of offloading place_pipeline's reactive OOM ladder had no strict_vram awareness: a CUDA OOM during placement would silently walk into model_offload even for a binding whose Resources(strict_vram=True) declares it cannot tolerate CPU-resident weights. Plan-time already refuses a PREDICTED offload under strict_vram, but a load-time OOM the plan didn't predict (e.g. a deploy-time binding that no longer matches the author's vram_gb sizing) reached the offload ladder anyway. Complements e4f9f6f's joint precision fit — defense in depth for any strict_vram binding, not just qwen-image's shared-component case. * th#1043: forced group-fit fp8 survives the bf16-resident upcast (0.48.1) (#375) Live 0.48.0 pod showed the joint fp8 forcing being silently un-done by load_from_pretrained's single-lane resident-upcast check (gw#534 rung 2): the first lane fit current free VRAM alone, upgraded back to bf16, and re-starved its sibling into the refused offload placement. A forced storage dtype now disables the local upgrade — the headroom belongs to the group, not the first lane to load. * th#1043: forced group-fit fp8 reports as a structural degradation (0.48.2) (#376) The joint shared-lane forcing served fp8 storage while the hub's plan said native bf16 — no FnDegraded, so placement never learns the card is too small for the group at native precision. Stamp the same rung outcome as the adaptive ladder. * th#1050: SDK lane contract — ctx.lane + opt-in handles= lane declaration (0.49.0) - ctx.lane: read-only executing lane on RequestContext, set post-degrade from the same truth as JobMetrics.lane; boot-warmup + CLI runs covered - @endpoint(handles=[...]): opt-in declared lane bodies (divergence marker, never inventory); validated against known concrete bodies; emitted into the discovery manifest as fn.handles - lane instruction: a declared lane is satisfiable with no laddered rebind (author code branches on ctx.lane); undeclared lanes refuse typed as before - hub-double e2e test: declared endpoint branches on ctx.lane over the real gRPC wire; JobMetrics.lane matches; undeclared dispatch unchanged * th#1051: RuntimeFormula SDK surface + worker-side term evaluation on the back-channel (0.50.0) * th#1051: uv.lock version bump * th#1051: lock gen-worker 0.50.0 * th#1051: mypy-clean Constant handling in formula eval/canonical * th#1055: desired-hot warm works on slot-only endpoints; refusals emit model events (0.50.1) ensure_desired_instance required the hot instance's binding set to equal spec.models, but deploy-bound Slots (ie#524/th#980) have no code default so spec.models is empty fleet-wide — every hub hot intent (gw#587 self-mint prewarm, th#912 seeding, #567 cell reload) failed a ValidationError that _reconcile_pass swallowed as one pod-local warning: no warmup, no self-mint, the th#868 w8a8 fence never opened, cells never armed. - validate against the declared slot set; code defaults may fill their own - declared-space hot bindings remap through the HelloAck precision picks - every desired-instance failure emits MODEL_STATE_FAILED for the instance refs (the silent stall is now impossible) * th#1055: version 0.50.2 (0.50.1 taken by pgw#626 on master) * th#1063: loud boot log when a datacenter pod has no CAS fill source (0.50.3) A RunPod pod without TENSORHUB_FILL_SOURCE_DIR (or with a non-mount path) pulls everything from R2 with write-through silently off — the defect that hid the fleet-wide unattached-volume regression. Boot now logs fill_source_disabled (reason=unset|not_a_mount) on datacenter pods and fill_source_enabled when the volume tier is live. * gw#627: Conv2d additive-branch support in the runtime LoRA overlay (0.51.0) Curated sdxl distill adapters (Lightning/DMD2) carry 49 conv LoRA pairs; the gw#547/558 branch was Linear-only, leaving raw peft as the only route on the w8a8 lane — fatal on _Fp8ScaledLinear (8/8 live turbo failures, th#1037 addendum). Plain nn.Conv2d denoiser modules are now branch-capable: canonical zeroed conv branches in the same rank bucket, staged-copy swap, eager instance-forward wrap conv1x1(conv(x,A),B). * pgw#628: residency reporting v2 — content-addressed idempotent observations (0.52.0) Worker half of th#1070: every applied HelloAck opens a republish epoch; the reconcile pass re-announces verified cached identities (exact ref+digest) once per epoch even when unchanged — a re-received plan (redrive/overdue resend/reconnect) is the hub asking for a resync, and success observations are safe to emit late, twice, or across plan revisions under the v2 hub's digest acceptance. Job-path cache hits within an epoch stay deduped. gw#614 same-set no-cancel unchanged. Revert-turns-red test; mypy+ruff clean; full suite 649 passed/5 skipped. No forced fleet rebuild — v2 hubs accept 0.44-0.51 reports fine. * gw#627 live fix: enable_compiled skips the branch lane on slot objects with no compile target A bare component slot (sdxl's AutoencoderKL vae) resolves none of cfg.targets; apply_lora_lane's no-denoiser error broke the whole model load (release-broken streak on the first gw627 deploy). Lane applies only to objects that resolve a compile target; the loud misconfig error stays for real targets. * gw#627 live fix 2: reject legacy attn-processor output from lora_state_dict normalization diffusers' non-diffusers converter turns kohya-flat sdxl attention keys into legacy …attn1.processor.to_q_lora.down.weight names that match no real module — the whole curated adapter failed typed (1680 unresolved keys, live request 88604466). Fall back to the raw keys, which resolve directly against module paths. gw627 test now runs the REAL SDXL converter in the loop. * th#1084 (pgw side): input-caused refusals raise ValidationError -> INVALID (0.53.0) RepoRefusal + SourceIncludeError are ValidationErrors; HF repo-access errors (RepositoryNotFound/Gated/Revision/EntryNotFound) wrap as typed input refusals at the convert ingest boundary with the class name preserved; an all-input-rejection 'no publishable flavor' aggregate is INVALID. Bad user repos fail only their request — never the release. * gw#631: delivered lora-bucket cell at a bare component slot is an ordinary miss, not a CellSelectionBugError r3 live chain (pod xzhlioakq909wq): the self-key check computed the family lora64 key with cfg.lora_bucket even when no cfg.target resolves on the slot object (sdxl's standalone vae), claimed the delivered cell as its own, and raised on lane drift (weight_lane 'lora64' != pipeline '') -> gw#608 seeded-cell self-mint refusal -> cell quarantine -> pod retired all_declared_functions_disabled. 0.52.1 downgraded apply_lora_lane for targetless slots; this applies the same effective-bucket rule to the self-key. * gw#632: renumber comments (id raced with slim-down filing) * th#1087 stages B+D: declared config params/envs (ConfigParam, env=), ctx.config, gen-push reconcile + local snapshot file for subprocesses (0.54.0) * th#1087: align worker config reconcile with final protocol * gw#631: remove unused worker and SDK surface * fix(config): pin subprocess snapshots per invocation * gw#631: retire unused H2D canary field * gw#496: remove silently dropped checkpoint metadata * gw#496: trim unused dataset conveniences * Project lifecycle shadow state from workers * Centralize worker lifecycle intent reporting * feat: report worker lifecycle intents * fix: restore RunJob org ownership contract * Implement exact config convergence protocol * marco-polo: regenerate uv.lock for editable gen-worker 0.56.0 * fix CI gates: type _hf_access_error_classes for except-clause mypy; drop unused Sequence import

* gw#587/gw#605: pin — a fresh Executor (no setup yet) advertises pre-load candidate cell lookups Unit-level the candidates work (records are eagerly created at construction); the LIVE gap (4 consecutive boots never received the stored cell — every boot re-minted) is therefore in the candidate->StateDelta->release-config-> attach chain, not in cell_lookups() itself. Forensics + eliminated theories in tracker gw#605. * compile-cache: platform org _system -> root (lockstep with tensorhub th#940 chaos 0e4f6b1e) system_repo now derives root/family-<f>; ref parsing, self-mint publisher owner, docs, proto contract, and test fixtures move in lockstep. Only the org segment changes — repo/flavor grammar is byte-identical. * gw#608: disable the AOT autograd cache in the compile path — cell portability requires the FX cache as the lookup surface; pack asserts its FX entries before publish ROOT CAUSE (cracked dry, three same-key captures from three pods): the AOTAutogradCache key hashes fx_kwargs[get_decomp_fn] via the function REPR — which embeds the process memory address (ASLR), so AOT keys can NEVER match across processes. inductor/fxgraph entries were bit-identical across all pods (8/8); inductor/aotautograd keys had zero overlap. On the consumer, the AOT-layer miss recompiles without consulting the portable on-disk FX entries: cache_hits=0, cache_misses=8 on two independent hosts against a byte-good cell (fail-closed held — never eager). Fix: _disable_aot_autograd_cache() (env TORCHINDUCTOR_AUTOGRAD_CACHE=0 + torch._functorch.config.enable_autograd_cache=False) at capture_env/ seed_env AND apply() — symmetric for producer capture and consumer seeding. Lookups collapse to FxGraphCache (proven portable); cells shrink (~50% was AOT payload); fresh processes pay a cheap AOT re-analysis, not an inductor recompile. Hardening (coordinator ruling): a minting boot proves its EXECUTION — finish_fleet_mint now also proves its ARTIFACT: refuse to publish a capture with zero fxgraph entries, and (when the proof supplies its compiled-graph count via finalize_self_mint(expected_graphs=pipe_misses)) a partial capture. Revert-turns-red tests for both. (5 pre-existing chaos test failures reproduced on untouched HEAD.) * gw#608: adopt-test fake warmup writes the fxgraph store the artifact assertion requires (one entry per miss) — the 0.40.4 CI failure was this un-updated fake, not the capture path * th#960/pgw#609 Phase 2: tests/harness/ + P1-P7 (stream lifecycle, residency reconcile, slot binding precedence, CAS multiwriter, discovery lock, cancel/ backpressure, boot smoke) Harness extracted per design: hub_double.py generalizes FakeScheduler from test_worker_grpc_e2e.py (the only double, at the true hub gRPC boundary); blob_host.py is a real blake3-verified HTTP blob server; toy_endpoints.py consolidates toy fixtures (echo/stream/GPU-slot-yield/Hub-bound/Slot-bound/ billable); subprocess_runner.py generalizes the gw#591 boot-smoke pattern. P3 absorbs PR #353 (pgw#606/th#938) and PR #307 (pgw#583) as real hub-double dispatches (no mocking of gen_worker internals, unlike the original unit tests) - revert-verified red against lifecycle.py's pre-fix state, restored clean. P1/P2/P4/P5/P6/P7 absorb assertions from test_worker_grpc_e2e.py, test_shared_cas_root_multiwriter.py, and test_boot_smoke_gw591.py. Found + fixed in the harness itself: WorkerHarness must set the TENSORHUB_CACHE_DIR env var (gen_worker.models.cache_paths reads the process-wide get_settings() singleton, not the per-worker Settings instance) - passing tensorhub_cache_dir= to load_settings() alone silently no-ops, and every hub-double test defaults to a fresh per-call temp dir so downloaded bytes never leak between tests. 40 passed, 1 skipped (pgw#605 heartbeat: proto fields not generated yet) in ~33s local. Existing test files not touched or deleted (Phase 3). * th#960/pgw#609 Phase 2: P8 (convert publish contract) + P9 (result upload metrics), plus ruff cleanup + a hub_double(file_base_url=) knob P9 needs P8 (tests/convert/test_p8_convert_publish_contract.py, hermetic fake_hub): dtype-mismatch cast honesty (pgw#589/th#901, condensed from test_publish_as_is_dtype.py), classifier corpus table (pgw#593), and an xfail(strict=True) test-first pin for pgw#566 (open bug: normalize_adapter _state_dict only passes unet_config to converters with a NAMED parameter - diffusers' real SDXL lora_state_dict takes **kwargs only, so the SGM block remap silently never runs). pgw#569's W8A8 verifier ULP gate is a documented skip - no factored-out comparator to test hermetically without a real w8a8 artifact round trip. P9 (tests/test_p9_result_upload_metrics.py): inline vs blob_ref by size alone over a real hub-double + real local media-upload dedup sink (no S3 part scripting, matching test_media_upload_owner.py's pattern); JobMetrics typed usage survives both wire forms. Needed hub_double(file_base_url=) so the sink URL rides the FIRST HelloAck instead of a racy post-connect mutate. ruff clean (unused imports, one E731 lambda->named helper). mypy: src/gen_worker clean (untouched); new tests/harness + test files carry the same untyped- fixture-param convention as the rest of tests/ (mypy only gates src/gen_worker per CI, tests/ was never in scope). Full new suite (P1-P9 + P8's own file): 50 passed, 1 skipped (pgw#605, proto fields not generated), 1 xfailed (pgw#566, fix open) in ~31s local. * th#960/pgw#609 Phase 3: delete 4 test files fully superseded by P3/P4/P8 test_slot_boot_precedence_th938.py, test_model_slot_identity_gate.py -> superseded by test_p3_slot_binding_precedence.py (same assertions, over a real hub-double boundary instead of monkeypatched Executor.store internals). test_shared_cas_root_multiwriter.py -> superseded by test_p4_cas_multiwriter_integrity.py (same 3 tests reproduced verbatim + 1 new row). tests/convert/test_publish_as_is_dtype.py -> superseded by tests/convert/test_p8_convert_publish_contract.py (same 3 core tests against the same fake_hub). Deletions only, no other changes. Full deletion manifest (all 166 files audited, reasoning for what's NOT deleted) is in the th#960/pgw#609 tracker checkpoint - the bulk of the pre-existing suite tests production surface (CLI, LoRA runtime, quantization lanes, VRAM/RAM admission, media/upload, Slot/discovery SDK, tests/convert producer suite) that no P-test covers, so it stays pending explicit direction on the larger absorb-or-accept-loss effort that would be needed to delete it safely. * pgw#610: proto — measured per-tier DiskUsageReport on StateDelta (lockstep with th#962) * pgw#610: measured per-tier disk telemetry in every StateDelta (th#962 lockstep) - statvfs on the real mounts (CAS root = container tier; ismount-guarded fill source volume when attached; NFS tier reserved) — never a declared size - safely-reclaimable = ref-index bytes of DISK-tier refs inactive AND not in the desired set (the disk-GC LRU's eligible set); reuses ref-index bytes, no tree rescans - capacity_generation bumps only on a measured (64 MiB-quantized) shape change; rides Hello.state and every StateDelta; 30s periodic refresher is edge-suppressed * pgw#609/th#960 Phase 4: rewire CI to one suite, one lane Manual dispatch + PR-into-master only (drafts/doc-only paths run nothing); drop the on-push trigger. Same gates (mypy gating, ruff non-gating per debt note, HTTP-timeout guard, llama-server, uv build); pytest targets the whole tests/ dir so P-suite additions land automatically. * Revert "pgw#609/th#960 Phase 4: rewire CI to one suite, one lane" This reverts commit c39efb69b82d967393bd3f3243ce328d954380f7. * th#960/pgw#609 Phase 2b: absorb ~26 incident-pinned rows into P1/P2/P3/P5/ P6/P8/P9 + 2 new consolidated files, ahead of the bounded aggressive deletion sweep Bucket-level triage per the coordinator's ruling (Paul's directive: minimal greenfield set, ~92% removal). Absorbed only distinct bug-classes carrying a real pgw#/gw# incident number or a revert-catches-it story; untagged "tested thoroughly once" coverage goes without absorption (git history is the archive). P1 (+6): auth-rejection exit/retry-within-window, permanent-precondition fast-exit, hello-ack-deadline reconnect, not-leader redirect, hub-restart reconnect - absorbed from test_worker_grpc_e2e.py (#372) ahead of its deletion. P2 (+3): setup-failure FnUnavailable+recovery, host-RAM-failure wire ordering (th#807) from test_worker_grpc_e2e.py; corrupt-load quarantine+ retry-once (gw#408, J17 flood) from test_snapshot_corruption.py. P3 (+1): bare-ref prefetch binding resolution (#377) from test_executor_prefetch_binding.py. P5 (+1): heavy-dep stub contract (pgw#506) from test_discovery_heavy_deps.py. P6 (+1): SendQueue capacity-generation fencing from test_worker_grpc_e2e.py. P8 (+5): diffusers root-allinone skip (e2e J7 ENOSPC), standalone-component canonical-weight selection (gw#426), transformers sharded-index onnx/pickle exclusion, gguf explicit-quant refusal, non-safetensors-format refusal family - from test_classifier.py ahead of its deletion. P9 (+1): token-bound-owner routing (J19 run34) from test_media_upload_owner.py. New consolidated files (coordinator-authorized, one per lane with no P-test home): tests/test_quantization_lanes.py (W8A8/W4A4/FP8 contract + gw#409 promote-device-integrity, from test_w8a8.py/test_w4a4.py/ test_fp8_and_emergency_loading.py/test_promote_device_integrity.py) and tests/convert/test_convert_producer_contract.py (clone concurrency gw#442, disk preflight gw#462, download-skip bank th#592, publish resilience gw#462, from the tests/convert/ producer suite). Also reverted a premature local-only Phase 4 CI commit (c39efb6, from a prior session before an interruption) - never pushed, out of THIS issue's authorized scope ("do not touch CI yet"); preserved in history via revert, not discarded. Full new-suite gates: mypy clean, ruff clean, each new/changed file green in isolation. Deletion commit (the ~125 files these rows supersede) follows as a separate, deletions-only commit per the coordinator's mechanics. * th#960/pgw#609 Phase 3b: aggressive deletion sweep — 125 files (32,618 LoC) superseded or absorbed per the coordinator's ruling Every file here is either fully superseded by test_quantization_lanes.py / test_convert_producer_contract.py, absorbed as rows into P1/P2/P3/P5/P6/P8/ P9 (chaos b8274a1), or carries no incident pin (git history is the archive, per the coordinator's bound). Full manifest, bucket-by-bucket reasoning, and the dangling-import sweep are in the th#960/pgw#609 tracker Checkpoint 6, pushed before this commit. Deletions only, no other changes. Foreign-WIP clusters untouched: compile_cache/executor_adopt/fleet_cells (this issue's own claim-checkpoint note) and test_disk_telemetry_pgw610.py (landed on chaos by another lane during this session). Final state: 14 test files (3,293 LoC) + tests/harness/ (6 files) + conftest.py x2 + fake_hub.py, target was ~12-15 files per the coordinator. * pgw#610: lint — typed enum cast + drop unused import * pgw#610: CONTRACT.md — document StateDelta.disk_usage / DiskUsageReport * pgw#610: guard drain's disk-report cancel with getattr (stubbed Lifecycles skip __init__, same convention as _cancel_residency_reconcile) * gw#608 residual: AOT-cache disable must be process-global — torch 2.13 config user overrides are thread-local ContextVars Live disproof (0.40.5, B200, 2026-07-21): mint capture still packed 8 ASLR-keyed aotautograd entries and the store-served sibling still failed (warmups=1, cache_hits=0, cache_misses=8) — the 0.40.4 disable ran on the arming thread; the warmup compile thread saw the default True. The env var is read once at torch import (env_name_force), so post-import it is inert. Fix: entrypoint sets TORCHINDUCTOR_AUTOGRAD_CACHE=0 before any torch import (binds all threads + compile-worker subprocesses); _disable_aot_autograd_cache additionally mutates the installed config entry's env_value_force (process- global) for already-imported-torch embedders. Cross-thread revert-turns-red test added (fails on the 0.40.5 shape, passes with the fix). Dry evidence: fxgraph keys 8/8 identical across ALL five banked packs (0.40.1 x3 / 0.40.3 / 0.40.5); aotautograd level-2 keys 0/8 everywhere. samples/selfmint-proof/gw0405-run/. * th#964: family lane policy twin — bare conv-UNet (sd1/sd2/sdxl) bindings AUTO-pick scale-free #fp8 locally gen_worker.models.ladder gains family_root (modelfamily.Root twin) and the conv-UNet w8a8 exclusion table; a maybe_rebind_family_fp8 fold (hub-parity AUTO pick over the resolve's sibling_flavors, sm_89+ only, bf16 stays the sub-floor default; pins untouched) runs in the local CLI resolve ahead of the GGUF fold over one shared resolve. WorkerResolvedRepo carries the resolve's model_family (slot-declared family as fallback); select_gguf no longer lets an AUTO-ineligible w8a8 row suppress the GGUF pick. Shared vectors: none in this repo; no format change. * gw#613/th#965: universal app-level heartbeat — layer 2 of the 3-layer liveness contract (0.41.0) Hello declares heartbeat_interval_ms=30000; the pgw#610 disk-report task is promoted to the beat: it force-re-sends the full StateDelta every 30s from the asyncio event loop (the control loop that owes progress, never a detached thread), starts at startup() entry so boot hangs are covered, and keeps beating through drain until the stream closes. Edge-triggered deltas still flow as before; the beat is the same message re-sent unchanged. Contract §3 rewritten to the 3-layer model (lockstep tensorhub c03a531d). Tests: hello cadence, force-send bypasses edge suppression, beats flow while a startup coroutine is parked (the gw#612 shape), drain keeps beating until close. * gw#613 retune: 10s beat x 6 misses; disk stats measured every 30s riding every beat HEARTBEAT_INTERVAL_MS 30000 -> 10000 (hub reaps at 6 misses, ~60s). The pgw#610 statvfs/ref-index scan keeps its 30s measurement cadence behind a TTL cache so the 10s beat never turns it into a hot loop; the report still rides every StateDelta. Contract §3 event-loop discipline clause added — already true in 0.41.0: setup/warmup/compile/residency/GC all run off-loop (executor._to_thread_complete / asyncio.to_thread). * gw#612: gate self-mint publish on full capture coverage; finalize phase; run-26 root cause corrected (0.42.0) Run 26's filed post-seal_publish hang is disproven: the worker completed setup and advertised readiness; the wedge was the hub's singular compile fence treating the 2-lane record's same-identity self-attested targets as ambiguous (tensorhub lockstep fix). Real worker defect fixed: a shared mint capture missing an unexercised mandatory sibling's graphs was still published as the family cell, bricking every adopting boot (gw#611 qwen hits=1/misses=1). finalize_self_mint packs only; the executor publishes iff every capture sharer proved in, else withholds typed+loud. New 'finalize' activity phase covers the post-proof tail to readiness. * gw#613: soften run-26 narrative — app-silent, not proven-hung (gw#612 root cause: healthy-idle, hub fence starvation) * gw#613: marco_polo_wedge — event-loop-blocking liveness probe for th#965 layer-2 drills * gw#611: credit AOT-layer hits in the adopt warmup proof; calls= in the fail-closed detail; cross-process portability repro Measured mechanism for th#954's release-bricking 0/0: bundled AOT cache serves a fresh-process adopt with fxgraph counters fully silent, so a HEALTHY serving cell read cache_hits=0/cache_misses=0 and fail-closed. inductor_counters now surfaces aot_cache_hit/aot_cache_miss and the guard wrapper counts AOT hits as serving evidence (AOT stays pinned off per gw#608; a config regression now degrades to a proven boot, not a brick). Fail-closed detail carries calls= to split orphaned-wrapper (calls=0) from counter-blind serving (calls>0). New tests: real mint->pack->fresh- subprocess adopt asserts >=1 FX hit; bundled-AOT 0/0 mechanism pin; guard credits AOT hits (red on revert). Hub lockstep: tensorhub chaos 9e7dca8e quarantines cells failing their own adopt proof. * th#767: gen_worker.families.wan — WanDefaults (animegen lane) Second family after SdxlDefaults: steps/guidance/guidance_2/max_guidance for the Wan 2.2 A14B MoE envelope. Registered under tensorhub's canonical architecture root "wan22" (not a Compile(family=) shape-grid string) so repo-metadata PUT validates against the matching tensorhub schema (companion wan22.schema.json on tensorhub chaos). Lets the wan-2.2 endpoint's T2V function migrate to the Slot model and give AnimeGen-T2V its own distilled recipe (8 steps, guidance=1.0, guidance_2=1.0) via repo metadata instead of branching on checkpoint identity. * gw#614: synthesized media-modality warmup coverage (union family cells publish) + on_hello_ack model-set-diff cancel (A) _run_synthesized_warmup coverage pass: compile-target objects still unexercised after the planned jobs get media VARIANTS of the same base payloads (warmup.media_variants: one optional image/audio field filled with a generated asset, nothing else drifts) — input-routed sibling lanes (qwen edit) mint into the shared capture, so the gw#612 gate publishes the union cell instead of withholding forever; adopts prove the sibling lane instead of arming unproven. (B) th#961 defense in depth: a HelloAck whose semantic model set (resolutions+disk_refs+snapshots+hot) matches the running reconcile's target no longer cancels it (in-flight self_mint_compile survives benign plan rewrites); changed sets cancel as before. * gw#615: fix event-loop-blocking disk telemetry — the 0.40.7 LTX post-seal_publish hang pgw#610/th#962's disk_usage_report() ran statvfs()/stat() synchronously inline in Lifecycle._state_delta(), which many hot paths call directly with no event-loop offload — including right after a self-mint's seal_publish. The provider-attached VOLUME fill-source is a network-backed mount that can stall for minutes under load (exactly what a self-mint's weight download + cell pack produce). A stalled statvfs() there freezes the whole event loop: every StateDelta, RunJob dispatch, and even the th#965 heartbeat (same loop) stops until the syscall returns. ModelStore.disk_usage_report() now only reads a cache; the actual measurement (_measure_disk_usage_report) runs exclusively via refresh_disk_usage_report(), off-loop through asyncio.to_thread. Lifecycle kicks that refresh as a fire-and-forget background task (never awaited inline), gated to the existing ~30s TTL, so a stalled mount delays only its own telemetry freshness — never any send, dispatch, or heartbeat. Root-caused via git diff v0.40.5..v0.40.7 (executor.py/lifecycle.py) plus gw#612's hub-log forensics for the qwen case: gw#612 itself was hub-side (dispatch-fence starve, corrected root cause, no worker hang) and its worker-side companion fix addresses a different defect (poisoned family-cell publish on the NEXT boot). This is a distinct, genuinely-new gen-worker regression introduced by a70e7c9 (wave-2 telemetry), reproduced locally with no GPU via a stalled-statvfs monkeypatch. Tests: tests/test_disk_telemetry_pgw610.py (new test_disk_usage_report_never_measures_only_reads_the_cache, test_state_delta_never_blocks_event_loop_on_a_stalled_mount; existing tests updated to the cache/refresh split) + tests/test_heartbeat_gw613.py (new test_beats_flow_while_disk_measurement_is_stalled; existing TTL test updated to the fire-and-forget refresh). All revert-turns-red verified. Full suite 458 passed/5 skipped; ruff+mypy clean. * gw#608: FX-key forensics — a store-served proof failure names the diverging FxGraphHashDetails component; compile_seconds + per-object proof counts in the fail-closed detail * gw#608: clamp fx-divergence detail under the 2000-char activity error cap * gw#608: exhaustive fx_cache_failure_report — counts + same-key re-save sibling diff + load probes discriminate B1/B2 in one boot * gw#608 FIX: self-mint arm is transactional over the process-global cache env — no-target siblings decline before any dir mutation, arm failure restores the prior env, and a delivered-cell-seeded process never opens a capture (the store-served 8/8-miss root cause) * gw#585: consume tensorhub v4 private-input manifests — v3->v4 hard cut (PR #320 port onto chaos) Proto adopted byte-for-byte from tensorhub chaos 07e5520b (protocol 4, RunJob.input_assets=15 — 14 taken by th#913 lane — InputAsset fields 1-6; CLI NDJSON stays v1); pb2 regenerated with grpcio-tools 1.82.1. Worker resolves the exact ordered manifest via one bounded resolve call under the attempt-scoped capability, verifies size/BLAKE3/MIME/kind/decode, preserves opaque refs, sets only local_path, attempt-owned temp cleanup; endpoint build rejects Asset-bearing set/frozenset and non-string maps; deterministic traversal (declaration order, list index, sorted string map keys). Full local suite 558 passed; mypy/ruff/ timeout-lint clean. uv.lock: mechanical 0.42.0 catch-up to pyproject. * gw#585: marco-polo-attach — private-input echo probe (blake3/size/local_path, optional in-flight delay) for the v4 e2e lane * gw#585: GEN_WORKER_INTERNAL_OBJECT_HOSTS — exact-host allowlist for resolver-minted private-input URLs (datacenter-internal object stores); caller public transports keep the full SSRF gate * gw#585: marco-polo-attach reads via os.fspath (Asset is os.PathLike; no .path() method) * gw#608: fix publish-event race in the self-mint adopt test fake — publish() fills the published dict before signaling the event (CI KeyError: 'bytes') * pgw#617: hierarchical slot bindings — load base composition, substitute components from ModelBinding.components - proto: ModelBinding.components (field 5, map component -> canonical ref; mirrors tensorhub th#980); pb regenerated - ModelRef.component_overrides (tensorhub-only, sorted pairs) — part of the binding identity, so a component-only rebind derives a new instance/ residency identity; flat bindings byte-identical - executor: _effective_spec stamps overrides per dispatch; setup materializes each override ref from its own snapshot and injects the loaded module via the gw#479 components= from_pretrained mechanism; pins/held refs/digests/compile-cell binding facts include override refs - loading.load_component_override: class from the BASE model_index.json, weights from the override tree's subtree/root - typed ComponentSubstitutionError (unknown component, non-CAS ref, non-worker-loaded slot) - registry: selected_by= slots may omit default_checkpoint (th#980 relaxation mirror; unblocks ie#524 sweep of wan-2.2/sdxl/z-image) - tests: hub-double flat->substitute->flat identity proof, typed unknown- component refusal, registry relaxation (564 green) * pgw#617: mypy — dedicated slot_comps var in _effective_spec * gw#619: report a typed hardware-unsuitable fact to the hub before the boot-time CUDA probe's silent exit entrypoint.py's boot sequence (gw#529 cuda_probe) logged GEN_WORKER_CUDA_PROBE_FAILED and exit(1)'d with zero orchestrator contact — invisible to every layer of telemetry (th#986: ~20 pod deaths whose cause, CUDA-13 images on old-driver hosts per th#988, was unattributable). On a probe failure the worker now dials the hub and sends ONE HardwareUnsuitable WorkerMessage in place of Hello (cozy.scheduler wire addition, tensorhub th#988 companion commit) before exiting — bounded best-effort (~7s worst case: 2 attempts x 3s + 1s backoff), never a reason to delay the exit materially, and a silent fallback when the hub is unreachable or predates this field (old-hub FAILED_PRECONDITION rejection). cuda_probe.py: classify_probe_failure() — typed reason_class vocabulary (torch_unavailable | cuda_unavailable | driver_too_old | cuda_error | unknown) derived from CudaProbeResult.reason; driver_too_old matches torch's own "driver too old (found version ...)" message verbatim (the exact th#591/th#979/th#988 signature). hardware_report.py (new): builds the report (nvidia-smi/NVML for driver_version+gpu_name — reads the driver directly, independent of the broken CUDA runtime being reported on; torch_version/torch_cuda_version; gen_worker_version; image_digest/instance_id from Settings; worker_id/ release_id from Settings or JWT claims) and dials it over a one-shot grpc.aio Connect stream. entrypoint.py: calls report_hardware_unsuitable() right where the probe failure was previously silently fatal; logs a cuda_probe_hardware_report startup phase either way. Tests: tests/test_hardware_report.py (classify_probe_failure vocabulary; build_hardware_report degrades safely without nvidia-smi and reads driver identity when torch.cuda is down; report_hardware_unsuitable delivers to a real hub-double socket, resolves identity from JWT claims, and returns promptly-false against both an old-hub protocol rejection and a connection-refused unreachable hub). tests/test_boot_smoke_hardware_report.py (true end-to-end: a real `python -m gen_worker.entrypoint` subprocess whose probe fails on THIS box's actual driver/torch mismatch dials a real hub socket, and the hub-unreachable fallback still exits without hanging). tests/harness/hardware_report_hub.py: shared real-socket test servicers (recording_hub, old_hub, closed_port_addr). Full suite green (578 passed) plus the extended gw#591 boot-smoke contract (tests/test_boot_smoke_gw591.py, tests/test_p7_boot_smoke.py) confirmed unaffected. * ie#522: non-inference functions must never report loading_functions() _warmup_plan already excludes non-"inference" kinds from boot warmup (spec.kind != "inference"), but available_functions()/loading_functions() didn't share that gate: a declared-but-never-dispatched conversion function (a release can bundle many; only a few get invoked) sat in loading_functions() forever, since nothing was ever going to warm it. The hub's th#965 layer-3 stall watchdog takes loading_functions() at face value and kills a worker after 10 minutes of "no open activity" on any name in that list — including a name nothing was ever loading. Reproduced live: a wan-2.2 conversion release bundling z-image-w8a8-quantization alongside clone-huggingface got its pod killed at 10m15s three times running, on three different hosts, each time mid a legitimate multi-GB clone-huggingface transfer. Fix: conversion/training/dataset-kind functions are always available (matching the existing spec.slots per-dispatch-serveability carve-out just above) and never counted as loading. * ie#522: watchdog evidence blind to I/O-bound download+load phases The activity watchdog's default liveness evidence was process CPU seconds only. Two real activities are honestly alive while burning near-zero CPU: an I/O-bound network model fill (large composites, tens of GB, CPU-light by design), and the subsequent on-disk load into GPU (safetensors mmap + device placement — third-party diffusers/torch code with no progress hook). Both read as "stalled" to the hub's th#965 layer-3 rule after 10 minutes of no heartbeat, even mid genuine progress. Reproduced live twice: identical ~9.5min self_mint_compile CancelledError at phase=load, two different RunPod hosts. Fixed three ways: - _process_cpu_evidence now sums live (not just reaped) child-process CPU via psutil — covers inductor's async_compile subprocess workers, which resource.RUSAGE_CHILDREN can't see while they're still running. - note_progress() lets an I/O callback (model-download byte ticks) heartbeat the current activity directly, independent of the CPU-sampling watchdog thread; wired into cozy_snapshot.py's _on_bytes network-byte path. - _process_io_evidence adds process disk I/O bytes (psutil io_counters) to the default combined evidence — a universal, non-invasive signal for the on-disk load phase that needs no app-level hook into third-party loading code at all. Revert-turns-red verified: reverting activity.py/cozy_snapshot.py makes exactly the 3 new evidence tests fail (zero-CPU download, zero-CPU disk load, live-child-CPU), while the pre-existing dead- activity-still-stalls test keeps the detector honest. Full suite green (584 passed, 5 skipped, 1 xfailed). * gw#621: progress registry — named monotonic counters on the 10s beat + self-diagnosis New gen_worker.progress: phases register named monotonic counters (download:<ref> bytes w/ snapshot-known totals, evidence:<kind> riding the watchdog, warmup:jobs, infer:steps, upload:bytes); family prefix maps to a per-phase self-diagnosis window (code constants). activity.on_beat rides lifecycle's 10s heartbeat: one counter-carrying ActivityUpdate per beat — frozen counters included (the hub kills on counter non-advancement) — and self_stalled=true + stalled_for_ms when even the freshest counter is past its window (kill on confession, never CPU inference). Additive ActivityUpdate fields 10-16 (counter, unit, done, total, rate, self_stalled, stalled_for_ms; th#994 carries the proto source). Counters are activity- owned (finished at activity end). Downloads now carry bytes_total on the FIRST DOWNLOADING event when the snapshot knows sizes — no more 'started total=0' then silence. Also completes ie#522's 7640551, which swept the in-flight activity.py half of this work into its commit — gen_worker.progress ships here. * gw#621: tests — registry windows/self-diagnosis, counter beats, hub-double e2e confession Revert-turns-red for the ie#522 false-positive class: a CPU-quiet byte-advancing download is visibly alive on the wire (counter beats, never self_stalled) over a REAL Worker + hub double; a frozen counter still confesses, and local behavior is unchanged (the hub owns the kill). * gw#621: progress_beat re-checks _done — a beat racing completion never re-opens a terminal activity hub-side * ie#522: arm_compile() no-ops (never raises) with no active arming scope Endpoint setup() declares intent by calling arm_compile() unconditionally (sdxl, wan-2.2, ...); whether that intent is active is the release's decision — an eager registration (no compile=Compile(...) declared) already opens ArmingScope(None, ...) as a documented no-op, but arm_compile() itself still raised RuntimeError when called with no active scope. Reproduced live: wan-2.2's eager smoke (dropped compile= to skip a 51.5min self-mint compile grid per Paul's ruling) crashed at warmup/phase=load on this exact raise. Fix: no active scope -> log at info, return False (never armed), same as every other "stays eager" path. Revert-turns-red verified (4 tests updated/added in test_arm_compile_pgw517.py). Full suite green (594 passed, 5 skipped, 1 xfailed). * pgw(th#1004): @variant_of handler annotation — variant pairing emitted into the discovery manifest (variant_of/variant), dangling/self/chain targets fail the build * ie#522: WanDefaults gains an optional shift field (scheduler flow-match shift) Live finding #3: the AnimeGen card's own 8-step recipe pins FlowMatchEulerDiscreteScheduler(shift=3.0) alongside its distillation LoRA -- a step-distilled lineage needs a non-default scheduler shift, which nothing in the wan22 inference-defaults vocabulary carried. shift: Optional[float] = None -- every non-distilled repo's metadata and the resolved-defaults fallback leave the pipeline's own scheduler config untouched; only a repo that explicitly publishes shift gets one applied at the endpoint's resolution site (wan-2.2 main.py, separate commit). Mirrors tensorhub-chaos's wan22.schema.json shift addition. * ie#522 fix: de-flake the zero-cpu watchdog test for constrained CI runners test_watchdog_survives_zero_cpu_download_via_note_progress was flaky by construction: in its 0.3s real-clock window the 5s note_progress rate-limit floor allowed exactly ONE beat, so the >=2 assertion silently depended on incidental watchdog CPU/IO evidence noise for the second beat — present on a 32-core dev box, absent on 2-core GitHub runners (confirmed by per-thread beat attribution: ticker contributes 1, the rest come from the watchdog thread; beats=1 reproduces even locally outside pytest). Deterministic rewrite, intent intact (zero-CPU download must not go silent while note_progress ticks arrive): drop the rate-limit floor for this test (the floor is covered by its own test), pin watchdog evidence to a constant zero so every counted beat is attributable to note_progress alone (airtight revert-turns-red — verified: neutering note_progress fails it with 0 beats), and wait event-driven on beat count with a 10s deadline instead of a fixed wall-clock window. Verified 20x unconstrained, 10x taskset -c 0, 5x taskset -c 0,1; full file green; mypy clean. * pgw#622: eager-while-compiling with hot-swap — novel request shapes serve immediately A novel input signature at a compiled target no longer stalls 30-60s behind Dynamo+Inductor: the guard routes it to the eager original, one background thread warms the compiled callable with a zero-filled dummy batch (separate CUDA stream, nice +10), and a successful warm atomically hot-swaps the signature to the compiled path. Each completed warm repacks the live cache root and republishes the cell under the same key so the fleet never compiles that (shape, GPU, lane) again. Sequential compile-then-serve is kept for: the boot warmup-proof window, mandatory quantized lanes (w8a8/w4a4), tight VRAM headroom (degrade, never OOM), regional mode, and signature-vocabulary explosions. * release: gen-worker 0.45.0 — pgw#622 eager-while-compiling with hot-swap * release: 0.45.0 uv.lock version sync * th#1017: SdxlScheduler gains lcm/euler_trailing distilled-regime schedulers * th#1017: @endpoint regimes= decorator surface + manifest emission Per-method {name: (regime,...)} dict on class handlers (mirrors warmup=), bare tuple on the function form, default ("standard",). Validated against {standard,v_prediction,distilled}; EndpointSpec.regimes carries the resolved per-handler tuple; discover.py emits "regimes" only when it differs from the default (absent = ["standard"]). * th#1017: ResolvedSlot.regime + executor-level regime backstop resolve_slot/resolve_slots thread inference_regime (default "standard") into ResolvedSlot.regime, sourced from the hub resolve response's inference_regime field (hub_client.WorkerResolvedRepo, mirrors model_family; "" on hubs not sending it). RegimeMismatchError raises when a resolved checkpoint's regime is outside the invoked function's declared regimes= — wired as a backstop in executor._resolve_slots_kwargs via allowed_regimes=spec.regimes. * th#1017: converter stamps regime-correct scheduler config run_clone/build_flavor_tree accept an inference_regime hint (payload field of the same name); writer.apply_regime_scheduler_config rewrites a produced diffusers flavor's scheduler/config.json: v_prediction sets prediction_type + rescale_betas_zero_snr, distilled sets trailing timestep_spacing. No-op for standard or a tree with no scheduler component. Updated the P8 build_flavor_tree test stub to accept the new kwarg. * th#1017: tests — regimes decorator surface, manifest, resolution, converter Real discovery/registry/slot/convert codepaths (no mocking): decorator validation errors, class/function manifest round-trip + default-omitted, ResolvedSlot.regime + RegimeMismatchError backstop, hub_client inference_regime parsing, and scheduler-config stamping for all three regimes. * gw#623: only cancel an in-flight residency load when the ACTIVE model leaves the set Root cause of the ~9.5min (and ~15s-burst) warmup/self_mint_compile CancelledError at phase=load on nearly every serving boot: _semantic_model_key hashed full snapshot serializations INCLUDING presigned file URLs, which the hub re-signs on every ~15s release-config rebuild and deliberately excludes from its own HelloAck semantic hash. Every resent ack carrying rotated URLs (or any sibling-ref churn, e.g. the th#1021 preposition/seed fight) looked like a model-set change and killed the in-flight load. - _snapshot_content_key: digest + (path, size, blake3) per file; URL excluded - level-triggered reconcile: passes re-read the freshest desired set; a benign mid-load update signals a re-pass instead of cancelling - cancel only when the actively-loading item left the set or changed snapshot identity / precision resolution (_active_work_still_desired) Tests (revert-turns-red verified): URL rotation keeps the load, sibling ref added mid-load keeps the load then converges, active ref leaving cancels, active-ref resolution repick cancels. * gw#624: free cancelled load/warmup attempts' memory before the retry Live: 5 cancelled load retries on one worker climbed container RAM 3%->97% and VRAM to 83.86GB (OOM on an 80GB card) — each cancelled attempt's partially loaded modules stayed alive (pinned by the CancelledError's traceback and by uncollected reference cycles) while the retry loaded a fresh copy on top. - _to_thread_complete: drop the joined Task reference on the cancel path so the discarded load result is not pinned by the traceback frame during rollback - _injection_kwargs: release shared-component holds acquired before the failure (they never reached rec.shared_keys, so vacate could not) - _rollback_failed_setup: gc.collect + cuda.empty_cache after vacate, and schedule an allocation purge - _setup_locked: run the purge (gc + empty_cache) under the load lock BEFORE the next attempt allocates — a retry provably starts from baseline Tests (revert-turns-red verified): cancelled _to_thread_complete result is unreachable while the exception lives; cancelled ensure_setup's buffers are gone at the next attempt's setup entry (gc disabled for the duration). * release: gen-worker 0.46.0 — th#1017 inference regimes * th#1031: cell_selection_bug recovers via self-mint instead of retry-blocking forever A self-requested cell whose graph signature drifts from this runtime's own (cell_key has no graph-shape axis, so structurally different sdxl graphs can share one key) used to raise CellSelectionBugError straight out of fleet_cells.enable_compiled, aborting the whole arm attempt. For a mandatory w8a8/w4a4 lane that becomes CompiledLaneUnavailableError, which fails setup entirely and gets retried from scratch on the next request/job -- hitting the identical stale delivered cell again, forever. Live pod_events proves this: 54 cell_selection_bug rows across 2 releases/9 pods over 24h+ (predating pgw#622 by ~12h, so unrelated to it), all rejecting one stale signature (c3c4395265ad) that nothing was ever able to overwrite. ArmOutcome now carries the caught CellSelectionBugError; enable_compiled falls through to self-mint (the ordinary MISS recovery) instead of raising. The th#883 invariant stays wire-visible -- callers still send the cell_selection_bug ModelEvent/pod_event -- it just no longer blocks serving. Self-mint's own fail-closed safety net (no CUDA/toolchain/target) is untouched, so a genuine mint impossibility on a mandatory lane still refuses, now chained from the selection bug via __cause__. Threaded through both compile-arming seams: the worker-loaded slot path (_enable_compiled) and the self-loaded pipeline path (ArmingScope/ arm_compile, what sdxl actually uses) via a new selection_bugs map so neither silently drops the alarm. * th#1043: joint precision fit for shared-component multi-lane loads Sequential per-lane loading let the first lane in a gw#479 shared- component group (qwen-image's t2i/edit) consume free VRAM at native precision, starving the second lane into an offload placement the shared-component invariant refuses outright ("shared-component lanes require resident placement"). Decide fp8 storage for the whole group up front, against its combined footprint, before any lane loads. * th#1043: strict_vram refuses a load-time CUDA OOM instead of offloading place_pipeline's reactive OOM ladder had no strict_vram awareness: a CUDA OOM during placement would silently walk into model_offload even for a binding whose Resources(strict_vram=True) declares it cannot tolerate CPU-resident weights. Plan-time already refuses a PREDICTED offload under strict_vram, but a load-time OOM the plan didn't predict (e.g. a deploy-time binding that no longer matches the author's vram_gb sizing) reached the offload ladder anyway. Complements e4f9f6f's joint precision fit — defense in depth for any strict_vram binding, not just qwen-image's shared-component case. * th#1043: forced group-fit fp8 survives the bf16-resident upcast (0.48.1) (#375) Live 0.48.0 pod showed the joint fp8 forcing being silently un-done by load_from_pretrained's single-lane resident-upcast check (gw#534 rung 2): the first lane fit current free VRAM alone, upgraded back to bf16, and re-starved its sibling into the refused offload placement. A forced storage dtype now disables the local upgrade — the headroom belongs to the group, not the first lane to load. * th#1043: forced group-fit fp8 reports as a structural degradation (0.48.2) (#376) The joint shared-lane forcing served fp8 storage while the hub's plan said native bf16 — no FnDegraded, so placement never learns the card is too small for the group at native precision. Stamp the same rung outcome as the adaptive ladder. * th#1050: SDK lane contract — ctx.lane + opt-in handles= lane declaration (0.49.0) - ctx.lane: read-only executing lane on RequestContext, set post-degrade from the same truth as JobMetrics.lane; boot-warmup + CLI runs covered - @endpoint(handles=[...]): opt-in declared lane bodies (divergence marker, never inventory); validated against known concrete bodies; emitted into the discovery manifest as fn.handles - lane instruction: a declared lane is satisfiable with no laddered rebind (author code branches on ctx.lane); undeclared lanes refuse typed as before - hub-double e2e test: declared endpoint branches on ctx.lane over the real gRPC wire; JobMetrics.lane matches; undeclared dispatch unchanged * th#1051: RuntimeFormula SDK surface + worker-side term evaluation on the back-channel (0.50.0) * th#1051: uv.lock version bump * th#1051: lock gen-worker 0.50.0 * th#1051: mypy-clean Constant handling in formula eval/canonical * th#1055: desired-hot warm works on slot-only endpoints; refusals emit model events (0.50.1) ensure_desired_instance required the hot instance's binding set to equal spec.models, but deploy-bound Slots (ie#524/th#980) have no code default so spec.models is empty fleet-wide — every hub hot intent (gw#587 self-mint prewarm, th#912 seeding, #567 cell reload) failed a ValidationError that _reconcile_pass swallowed as one pod-local warning: no warmup, no self-mint, the th#868 w8a8 fence never opened, cells never armed. - validate against the declared slot set; code defaults may fill their own - declared-space hot bindings remap through the HelloAck precision picks - every desired-instance failure emits MODEL_STATE_FAILED for the instance refs (the silent stall is now impossible) * th#1055: version 0.50.2 (0.50.1 taken by pgw#626 on master) * th#1063: loud boot log when a datacenter pod has no CAS fill source (0.50.3) A RunPod pod without TENSORHUB_FILL_SOURCE_DIR (or with a non-mount path) pulls everything from R2 with write-through silently off — the defect that hid the fleet-wide unattached-volume regression. Boot now logs fill_source_disabled (reason=unset|not_a_mount) on datacenter pods and fill_source_enabled when the volume tier is live. * gw#627: Conv2d additive-branch support in the runtime LoRA overlay (0.51.0) Curated sdxl distill adapters (Lightning/DMD2) carry 49 conv LoRA pairs; the gw#547/558 branch was Linear-only, leaving raw peft as the only route on the w8a8 lane — fatal on _Fp8ScaledLinear (8/8 live turbo failures, th#1037 addendum). Plain nn.Conv2d denoiser modules are now branch-capable: canonical zeroed conv branches in the same rank bucket, staged-copy swap, eager instance-forward wrap conv1x1(conv(x,A),B). * pgw#628: residency reporting v2 — content-addressed idempotent observations (0.52.0) Worker half of th#1070: every applied HelloAck opens a republish epoch; the reconcile pass re-announces verified cached identities (exact ref+digest) once per epoch even when unchanged — a re-received plan (redrive/overdue resend/reconnect) is the hub asking for a resync, and success observations are safe to emit late, twice, or across plan revisions under the v2 hub's digest acceptance. Job-path cache hits within an epoch stay deduped. gw#614 same-set no-cancel unchanged. Revert-turns-red test; mypy+ruff clean; full suite 649 passed/5 skipped. No forced fleet rebuild — v2 hubs accept 0.44-0.51 reports fine. * gw#627 live fix: enable_compiled skips the branch lane on slot objects with no compile target A bare component slot (sdxl's AutoencoderKL vae) resolves none of cfg.targets; apply_lora_lane's no-denoiser error broke the whole model load (release-broken streak on the first gw627 deploy). Lane applies only to objects that resolve a compile target; the loud misconfig error stays for real targets. * gw#627 live fix 2: reject legacy attn-processor output from lora_state_dict normalization diffusers' non-diffusers converter turns kohya-flat sdxl attention keys into legacy …attn1.processor.to_q_lora.down.weight names that match no real module — the whole curated adapter failed typed (1680 unresolved keys, live request 88604466). Fall back to the raw keys, which resolve directly against module paths. gw627 test now runs the REAL SDXL converter in the loop. * th#1084 (pgw side): input-caused refusals raise ValidationError -> INVALID (0.53.0) RepoRefusal + SourceIncludeError are ValidationErrors; HF repo-access errors (RepositoryNotFound/Gated/Revision/EntryNotFound) wrap as typed input refusals at the convert ingest boundary with the class name preserved; an all-input-rejection 'no publishable flavor' aggregate is INVALID. Bad user repos fail only their request — never the release. * gw#631: delivered lora-bucket cell at a bare component slot is an ordinary miss, not a CellSelectionBugError r3 live chain (pod xzhlioakq909wq): the self-key check computed the family lora64 key with cfg.lora_bucket even when no cfg.target resolves on the slot object (sdxl's standalone vae), claimed the delivered cell as its own, and raised on lane drift (weight_lane 'lora64' != pipeline '') -> gw#608 seeded-cell self-mint refusal -> cell quarantine -> pod retired all_declared_functions_disabled. 0.52.1 downgraded apply_lora_lane for targetless slots; this applies the same effective-bucket rule to the self-key. * gw#632: renumber comments (id raced with slim-down filing) * th#1087 stages B+D: declared config params/envs (ConfigParam, env=), ctx.config, gen-push reconcile + local snapshot file for subprocesses (0.54.0) * th#1087: align worker config reconcile with final protocol * gw#631: remove unused worker and SDK surface * fix(config): pin subprocess snapshots per invocation * gw#631: retire unused H2D canary field * gw#496: remove silently dropped checkpoint metadata * gw#496: trim unused dataset conveniences * Project lifecycle shadow state from workers * Centralize worker lifecycle intent reporting * feat: report worker lifecycle intents * fix: restore RunJob org ownership contract * Implement exact config convergence protocol * marco-polo: regenerate uv.lock for editable gen-worker 0.56.0 * th#1098: marco-polo example 1.0.1 — gen-worker range to >=0.56,<0.57 (protocol v5), relock * pgw#636/#637/#638: hot-GPU mandate — pack VRAM with checkpoints, credit dynamo's in-memory cache, stop starving staged downloads pgw#637 (P0, the 2026-07-24 sdxl root cause). Cell keys are checkpoint-free, so the 2nd checkpoint of an already-proven family serves its warmup from dynamo's in-memory compiled code with ZERO FX/AOT counter movement (calls=1, hits=0, misses=0) and was deterministically judged "disproven" -> CompiledLaneUnavailableError -> FnUnavailable compile_cell_failed on every worker. The finalize proof now credits that signature, gated on BOTH: 1. compile_cache.cell_proven_in_process(ref) — this process actually served that exact cell (a real FX/AOT hit, or a finalized self-mint), and 2. compile_cache.has_inmemory_compiled_code(pipe) — DIRECT evidence from torch._dynamo that compiled code is live for THIS object's targets. Both are load-bearing: (1) alone lets a sibling object's hit certify this object's silence, which gw#603/gw#611 forbid (and which broke two guard tests); (2) alone would credit a cell never proven anywhere. Aggravator fixed: apply() tracks armed pipelines in a WeakSet and unwrap() skips the PROCESS-GLOBAL torch._dynamo.reset() while a sibling is still armed, so a failing 2nd arm no longer kills the healthy 1st checkpoint's lane. pgw#636 (Paul: "vram_gb means 'at least X', not 'only X'"). The one-pipeline ceiling was the pre-load ask: max(sum(vram_hints), Resources.vram_gb) reserved 12 GiB (+2 GiB margin) for every never-seen checkpoint pick and evicted the resident pipeline on every hop. _estimate_setup_need now prefers a prior MEASURED vram_hint, else the wire snapshot's real byte total, and falls back to the declaration only when a ref has no byte facts at all. Slot( share_components=...) opts a dynamic slot into cross-pick content-keyed component sharing (gw#479 generalized, content-honest: unequal bytes never alias). Residency shared entries move from refcount to `holders`: referenced components are demotable while idle (owners pin+promote before executing), never evictable while referenced, and 2+-holder entries sort LAST in the LRU; record vacate no longer eagerly drains them. pgw#638. Disk staging is network+disk I/O and holds no GPU lock, so _reconcile_pass no longer gates it on executor.wait_idle() and on_message(run_job) no longer cancels it (only an in-flight HOT warm is preempted). Live: staged 4.7 GB downloads sat at 0% for 4+ minutes on busy workers while the hub bought pods for models the fleet nearly had. The reconcile loop parks on wait_idle() OR restart after a pass on a busy worker, so a fresh HelloAck is never stuck behind a long tenant job. Tests (real code paths; fakes only at the torch/network/capacity boundary): - test_compile_proof_pgw637.py — proven-cell registry, the dynamo probe (negative and positive), unwrap sibling-scoping. - test_executor_adopt.py::test_second_checkpoint_served_from_dynamo_inmemory_cache_proves — the filed regression through the REAL ensure_setup finalize path, plus the same counter signature WITHOUT live dynamo code still failing closed. - test_hot_gpu_pgw636.py — real _estimate_setup_need, real Residency holders semantics and LRU ordering, the real make_room packing juggle (four ~5 GB picks hot on a 24 GiB budget, one LRU victim under real pressure, warm promote back), serve-while-downloading over the real worker + hub-double, and the end-to-end juggle: four checkpoints through ONE selected_by= slot, round two runs zero additional setups (all four instances retained). * pgw#639 (partial): SIGUSR2 -> faulthandler stack dump of every thread The worker's asyncio loop owns transport and the 10s heartbeat while all model work runs on threads, so a thread wedged inside a C extension keeps heartbeating and hub-side logs can never see it. Registering faulthandler on SIGUSR2 at boot makes 'kill -USR2 <pid>' print all thread stacks to stderr, allocation-free, so it also works under memory pressure. This is the filed 'trivial first step'. The delivery half (getting a signal into the pod: sshd behind an env flag, or a hub-mediated debug RPC) is still open on pgw#639. * th#1107: three serve-path fixes — vae_only stops taxing pipelines that fit, strict_vram binds the auto SELECTION, image path signals the finalize handoff vae_only is chosen when the pipeline FITS and only headroom is tight, yet it enabled VAE tiling + attention slicing (memory.py:672) — tiled decode re-runs the VAE per tile and attention slicing replaces the fused SDPA/flash kernel. Both are now reserved for the rungs where the model genuinely does not fit. strict_vram was enforced only inside the OOM-demotion branch, so select_auto_mode returning model_offload/group_offload placed CPU-resident weights with no OOM and no error — the th#1043 shape reaching serving as a silent 5-10x tax. write_image never called _release_gpu_slot_for_finalize, so gw#516's overlap was dead on every image endpoint: finalize_wall_ms == 0 on 19 of 20 endpoints in the master hub DB (979 sdxl samples), non-zero only on ltx-video (median 3.25s of a 14.9s request). The encode+upload tail now runs slotless like the video path. * gw#640/th#1077: worker fatals reach the hub, not just pod stdout entrypoint._log_worker_fatal wrote the exception class, message and traceback to the pod's stdout ONLY. RunPod exposes no container-logs API, so a cloud-only worker death was un-debuggable by construction — the th#1085 cold-boot investigation burned six live runs on a crash whose traceback existed and was unreachable. The fatal is now also dialed to the orchestrator over a fresh Connect stream, reusing the HardwareUnsuitable carrier with reason_class="worker_fatal". The hub already persists that message as a durable pod_events row (class + reason + full JSON payload), so this needs no proto change and no hub redeploy, and works against every hub pin already deployed. Second half: the run loop ending WITHOUT a hub Drain or a shutdown signal was a clean, silent `return 0` — exactly the gw#640 signature the hub could only observe as a young-worker death. That is now a reported UnexpectedWorkerExit fatal. Worker.run() keeps its int contract and reports from the sync wrapper; the cuda_probe path is left alone because it already sends its own typed HardwareUnsuitable report. * pgw#638: revert serve-while-downloading — the naive fix breaks the mutable-tag identity fence Dropping the tenant-idle gate on disk staging (and the run_job cancellation of the reconcile pass) is the obvious fix for the measured starvation, and it is wrong. The pass's first act for a ref is to fence a stale resident identity via revalidate_snapshot_identity; a tenant job may legitimately be re-materializing OLDER bytes for that same ref concurrently. Unserialized, the fence loses the race and the worker reports the stale identity indefinitely — the th#1066 hub/worker residency drift class. Evidence: test_p2_residency_reconcile.py:: test_mutable_tag_move_fences_events_by_digest_and_generation goes red and does not converge within 10s of polling. Two attempted patches also failed: - re-verify after every job — handle_run_job returns BEFORE the job's load completes, so the re-verify still races it; - gate only when a fence is actually pending (needs_identity_fence) — the predicate is evaluated before the job's record becomes ready, so it reads False and skips the gate exactly when it is needed. The correct fix is pgw#641 Stage 4: transfers become their own tasks with their own lifecycle, so staging never sits inside this serialized pass and never has to choose between progress and ordering. Recorded in pgw#638. pgw#636 and pgw#637 are unaffected and stay. Also drops the `hold` toy endpoint and the serve-while-downloading test that only existed for the reverted behavior; the four-checkpoint juggle test stays. * pgw#650: legacy sweep — delete the pre-slot ModelChoice binding vocabulary, the dead ar_tts runtime, and 12 zero-consumer symbols The model SET is catalog, not code (th#980/ie#524/th#1087): Model(id, Hub(...), defaults) rows embedded a curated checkpoint list in endpoint code, superseded by Slot(selected_by=) + hub mapping. Zero endpoints declare one; the model.choices[] manifest block has never been emitted in production. - delete api/model.py (Model/ModelChoice/ModelDefaults/is_model_choice) and discovery's _model_choice_in/_collect_model_placement_key + fn["model"] block - delete runtimes/ar_tts.py: zero importers, zero tests, zero docs, zero consumers; runtimes/__init__ advertised it in __all__ without importing it - delete 12 symbols with no reference in src, tests, or any consumer repo: resolve_slots, DestinationRepo, read_gguf_metadata, materialize_safetensors_input, is_calibration_required, is_inline_supported, spec_id, snapshot_weight_bytes, with_oom_retry, quantized_modules, content_credentials._reset_for_tests - tests: delete test_request_context_checkpoint_contract.py (inspect.signature only, executes no behavior), test_boot_smoke_gw591.py (assertion-identical duplicate of test_p7_boot_smoke.py, pays a second real torch subprocess boot per run), test_families_wan.py (WanDefaults has no consumer; resolve_slot precedence is covered by P3), test_testing_helpers.py (tests the test helper); drop two rows that assert a protobuf setter and an exception's own f-string suite: 725 passed, 5 skipped, 1 xfailed. ruff + mypy clean on src. * gw#640: post-mortem supervisor — report the death that happens below Python 0.56.1 instruments every death Python can see; run 9 produced six restarts and zero worker_fatal, so the remaining class is a signal (cgroup OOM SIGKILL, SIGSEGV, external kill). Fork a supervisor before the heavy imports: the parent stays tiny, outlives the worker, and reports WTERMSIG/WCOREDUMP plus memory.max/current/peak and the memory.events oom_kill delta through the existing worker_fatal carrier. A boot record covers whole-cgroup kills — the next boot reports the previous container death. Also: the host canary reported os.cpu_count() (the HOST's 32) next to a cgroup-derived ram_total_gb (the container's 14.9 GB). Report the cores this container may actually use, and benchmark with that many threads. * gw#640: prove the supervisor forwards SIGTERM (drain semantics unchanged) * pgw#648/#641-S2/#647-C1: per-device-group VRAM accounting, admission leases, single-flight per instance Three trains of the residency-v2 program (WORKER-RESIDENCY-DESIGN.md is the design of record). All real planner/admission/eviction logic under test; the only fake is torch.cuda (th#1105 doctrine). A. pgw#648 — DeviceGroup is the unit of placement. `_default_free_vram_bytes` SUMMED free VRAM across every CUDA device, so a 3x24GB pod reported 72GB free and admitted a 30GB model that fits on no single card. VRAM is not fungible between cards. `DeviceGroup(devices=...)` now owns the probe; one `Residency` accounts for exactly one group's pool and nothing here ever sums across groups. A group spanning devices (future TP mesh) sums its own members only — that is one placement unit by definition. B. pgw#641 Stage 2 — admission leases. `Residency.admit(sizes) -> Lease` replaces the gw#409 whole-job `executing()` pin on the run path. From admission on: (a) no eviction/demotion/release path may victim a leased ref — INCLUDING refs whose entries do not exist yet, which is the structural gap the pin no-op'd on (a freshly created entry was demotable between its track_vram and the execution-time pin); (b) bytes for not-yet-loaded refs are RESERVED, so two concurrent admissions can no longer book the same free VRAM and OOM each other mid-load. Two leases on one ref claim the MAX, not the sum — one future load, one claim — and a claim is consumed the moment track_vram books the real bytes. `make_room(for_refs=)` excludes the caller's own reservations (they are the demand being satisfied). `fits(sizes)` is the cheap honest "can this worker serve this now?" query the hub can eventually route on. Expected bytes follow the pgw#636 ask ladder: measured hint -> this dispatch's snapshot total -> banked snapshot -> 0. Admission never refuses here; the adaptive-fit ladder still owns overcommit. C. pgw#647 (C1) — single-flight per instance. One live instance == one binding set == one materialized graph with MUTABLE buffers (the resident LoRA branch; adapter enable state). Two concurrent requests on one instance corrupt each other. One-job-per-GPU masked this; `BoundedSemaphore(gpu_count)` on multi-GPU pods already does not. A per- _ClassRecord run gate now serializes adapter attach + handler + detach; jobs on DIFFERENT instances (different picks) still run concurrently. Acquired after the GPU permit and never held while acquiring one, so no cycle. `@endpoint(reentrant=True)` is the explicit opt-out for classes that mutate no instance state; it is a class-only declaration (a loose function holds no instance state to serialize) and rejects loudly on functions. tests/test_residency_v2.py is the program's one designed test home (pgw#645); later stages add rows there rather than new per-issue files. Verified in an isolated `git archive HEAD` overlay (the shared chaos worktree has 2+ other lanes mid-edit): 725 passed / 5 skipped / 1 xfailed, ruff + mypy clean. * th#1111: split runtime_ms into real stages — measured, classified, and reconciling runtime_ms was one opaque number over input fetch, text encode, denoise, VAE decode, image encode, credential stamp and upload; the GPU-permit wait was in no metric at all. Four fleet benchmarks ran without attributing a second. - stage_timing.py: per-request StageTimer. Exclusive nesting, per-thread stacks (uploads fan out), an interval log for window attribution. Renders JobMetrics.stage_ms with measured stages + total.prep / total.tail / total.denoise / class.{gpu_busy,small_gpu,gpu_idle,unattributed} / resid.{prep,tail,unattributed,overlap}. slot_prologue names the gap between runtime_ms's start and the handler window, so the map sums to runtime_ms EXACTLY rather than approximately. - denoise measured with no endpoint change: diffusers_step_callback marks every step end (un-throttled — the emit stays throttled), giving denoise total + denoise.step_mean on all 14 endpoints wired to the shared callback. - framework hooks: gpu_permit_wait and input_fetch (pre-handler, reported but outside the reconciliation), image_encode, video_encode, credential_stamp, upload (promotes _stream's finalize_elapsed_s). - ctx.stage("text_encode") for what only an endpoint knows. - class.gpu_busy / total.runtime is the per-request half of pgw#652's hot-fraction metric. - proto: JobMetrics.stage_ms = 15, additive; the vocabulary is documented at the field. Integration test drives the real worker over the hub-double with an image-shaped handler; the map reconciles with the wire's runtime_ms (675ms sample: text_encode 60 + denoise 161 + image_encode 386 + stamp 1 + resid.unattributed 67 = 675, exact). Un-bracketed decode time surfaces as resid.tail instead of vanishing. Unit tests cover the permit wait, exclusive nesting, the prologue and the concurrent-upload overlap case. * gw#640: dial the hub only for the deaths Python could not report (signal / whole-container), and isolate the boot record per boot-smoke run An ordinary non-zero exit already had 0.56.1's in-process worker_fatal, so the supervisor re-dialing it duplicated the row and added the ~7s report budget to every failing boot — which is what tipped the boot-smoke harness over its 25s budget. The shared default boot-record path also made one test report the previous test's death. * pgw#652: activation-aware admission — leases reserve the workspace a concurrent request allocates, learned from measured peaks Weights are not the whole cost of admitting a request. A 1024^2 diffusion request also holds GBs of latents and attention workspace, and the design's own warning is that "leases must reserve activation headroom PER CONCURRENT REQUEST or admission control will OOM the moment interleaving starts working". This builds that on the pgw#641 Stage 2 leases. Accounting. A `Lease` now carries an activation claim. Weight claims MAX per ref (two jobs needing one checkpoint share ONE future load); activation claims SUM across live leases (each concurrent request allocates its own workspace), and are never consumed — only released — because they are transient by definition. `fits()` counts the caller's own activation alongside its weights. `make_room()` subtracts OTHER in-flight requests' activation but not the calling lease's: that one is either already allocated (hence already visible in the measured free bytes) or is this job's own future demand, and double-counting it would evict a resident checkpoint on every load — exactly the pgw#636 pathology this program just removed. Learned, never declared (standing rule 2: no developer-facing residency knobs). `record_activation` takes `peak_vram_bytes` minus what was already allocated when the handler took the GPU — a measurement the executor already had in hand and threw away — and keeps a DECAYING high-water: up instantly in full (under-reserving is what OOMs) and bleeding 12.5% per subsequent request otherwise, so one 4096^2 outlier does not permanently tax residency depth for a stream of 512^2 requests. It is keyed by FUNCTION, not by pick: transient VRAM is a property of the shape and the graph, not of which checkpoint is bound, so a never-seen checkpoint inherits a real measurement instead of rese…

* gw#587/gw#605: pin — a fresh Executor (no setup yet) advertises pre-load candidate cell lookups Unit-level the candidates work (records are eagerly created at construction); the LIVE gap (4 consecutive boots never received the stored cell — every boot re-minted) is therefore in the candidate->StateDelta->release-config-> attach chain, not in cell_lookups() itself. Forensics + eliminated theories in tracker gw#605. * compile-cache: platform org _system -> root (lockstep with tensorhub th#940 chaos 0e4f6b1e) system_repo now derives root/family-<f>; ref parsing, self-mint publisher owner, docs, proto contract, and test fixtures move in lockstep. Only the org segment changes — repo/flavor grammar is byte-identical. * gw#608: disable the AOT autograd cache in the compile path — cell portability requires the FX cache as the lookup surface; pack asserts its FX entries before publish ROOT CAUSE (cracked dry, three same-key captures from three pods): the AOTAutogradCache key hashes fx_kwargs[get_decomp_fn] via the function REPR — which embeds the process memory address (ASLR), so AOT keys can NEVER match across processes. inductor/fxgraph entries were bit-identical across all pods (8/8); inductor/aotautograd keys had zero overlap. On the consumer, the AOT-layer miss recompiles without consulting the portable on-disk FX entries: cache_hits=0, cache_misses=8 on two independent hosts against a byte-good cell (fail-closed held — never eager). Fix: _disable_aot_autograd_cache() (env TORCHINDUCTOR_AUTOGRAD_CACHE=0 + torch._functorch.config.enable_autograd_cache=False) at capture_env/ seed_env AND apply() — symmetric for producer capture and consumer seeding. Lookups collapse to FxGraphCache (proven portable); cells shrink (~50% was AOT payload); fresh processes pay a cheap AOT re-analysis, not an inductor recompile. Hardening (coordinator ruling): a minting boot proves its EXECUTION — finish_fleet_mint now also proves its ARTIFACT: refuse to publish a capture with zero fxgraph entries, and (when the proof supplies its compiled-graph count via finalize_self_mint(expected_graphs=pipe_misses)) a partial capture. Revert-turns-red tests for both. (5 pre-existing chaos test failures reproduced on untouched HEAD.) * gw#608: adopt-test fake warmup writes the fxgraph store the artifact assertion requires (one entry per miss) — the 0.40.4 CI failure was this un-updated fake, not the capture path * th#960/pgw#609 Phase 2: tests/harness/ + P1-P7 (stream lifecycle, residency reconcile, slot binding precedence, CAS multiwriter, discovery lock, cancel/ backpressure, boot smoke) Harness extracted per design: hub_double.py generalizes FakeScheduler from test_worker_grpc_e2e.py (the only double, at the true hub gRPC boundary); blob_host.py is a real blake3-verified HTTP blob server; toy_endpoints.py consolidates toy fixtures (echo/stream/GPU-slot-yield/Hub-bound/Slot-bound/ billable); subprocess_runner.py generalizes the gw#591 boot-smoke pattern. P3 absorbs PR #353 (pgw#606/th#938) and PR #307 (pgw#583) as real hub-double dispatches (no mocking of gen_worker internals, unlike the original unit tests) - revert-verified red against lifecycle.py's pre-fix state, restored clean. P1/P2/P4/P5/P6/P7 absorb assertions from test_worker_grpc_e2e.py, test_shared_cas_root_multiwriter.py, and test_boot_smoke_gw591.py. Found + fixed in the harness itself: WorkerHarness must set the TENSORHUB_CACHE_DIR env var (gen_worker.models.cache_paths reads the process-wide get_settings() singleton, not the per-worker Settings instance) - passing tensorhub_cache_dir= to load_settings() alone silently no-ops, and every hub-double test defaults to a fresh per-call temp dir so downloaded bytes never leak between tests. 40 passed, 1 skipped (pgw#605 heartbeat: proto fields not generated yet) in ~33s local. Existing test files not touched or deleted (Phase 3). * th#960/pgw#609 Phase 2: P8 (convert publish contract) + P9 (result upload metrics), plus ruff cleanup + a hub_double(file_base_url=) knob P9 needs P8 (tests/convert/test_p8_convert_publish_contract.py, hermetic fake_hub): dtype-mismatch cast honesty (pgw#589/th#901, condensed from test_publish_as_is_dtype.py), classifier corpus table (pgw#593), and an xfail(strict=True) test-first pin for pgw#566 (open bug: normalize_adapter _state_dict only passes unet_config to converters with a NAMED parameter - diffusers' real SDXL lora_state_dict takes **kwargs only, so the SGM block remap silently never runs). pgw#569's W8A8 verifier ULP gate is a documented skip - no factored-out comparator to test hermetically without a real w8a8 artifact round trip. P9 (tests/test_p9_result_upload_metrics.py): inline vs blob_ref by size alone over a real hub-double + real local media-upload dedup sink (no S3 part scripting, matching test_media_upload_owner.py's pattern); JobMetrics typed usage survives both wire forms. Needed hub_double(file_base_url=) so the sink URL rides the FIRST HelloAck instead of a racy post-connect mutate. ruff clean (unused imports, one E731 lambda->named helper). mypy: src/gen_worker clean (untouched); new tests/harness + test files carry the same untyped- fixture-param convention as the rest of tests/ (mypy only gates src/gen_worker per CI, tests/ was never in scope). Full new suite (P1-P9 + P8's own file): 50 passed, 1 skipped (pgw#605, proto fields not generated), 1 xfailed (pgw#566, fix open) in ~31s local. * th#960/pgw#609 Phase 3: delete 4 test files fully superseded by P3/P4/P8 test_slot_boot_precedence_th938.py, test_model_slot_identity_gate.py -> superseded by test_p3_slot_binding_precedence.py (same assertions, over a real hub-double boundary instead of monkeypatched Executor.store internals). test_shared_cas_root_multiwriter.py -> superseded by test_p4_cas_multiwriter_integrity.py (same 3 tests reproduced verbatim + 1 new row). tests/convert/test_publish_as_is_dtype.py -> superseded by tests/convert/test_p8_convert_publish_contract.py (same 3 core tests against the same fake_hub). Deletions only, no other changes. Full deletion manifest (all 166 files audited, reasoning for what's NOT deleted) is in the th#960/pgw#609 tracker checkpoint - the bulk of the pre-existing suite tests production surface (CLI, LoRA runtime, quantization lanes, VRAM/RAM admission, media/upload, Slot/discovery SDK, tests/convert producer suite) that no P-test covers, so it stays pending explicit direction on the larger absorb-or-accept-loss effort that would be needed to delete it safely. * pgw#610: proto — measured per-tier DiskUsageReport on StateDelta (lockstep with th#962) * pgw#610: measured per-tier disk telemetry in every StateDelta (th#962 lockstep) - statvfs on the real mounts (CAS root = container tier; ismount-guarded fill source volume when attached; NFS tier reserved) — never a declared size - safely-reclaimable = ref-index bytes of DISK-tier refs inactive AND not in the desired set (the disk-GC LRU's eligible set); reuses ref-index bytes, no tree rescans - capacity_generation bumps only on a measured (64 MiB-quantized) shape change; rides Hello.state and every StateDelta; 30s periodic refresher is edge-suppressed * pgw#609/th#960 Phase 4: rewire CI to one suite, one lane Manual dispatch + PR-into-master only (drafts/doc-only paths run nothing); drop the on-push trigger. Same gates (mypy gating, ruff non-gating per debt note, HTTP-timeout guard, llama-server, uv build); pytest targets the whole tests/ dir so P-suite additions land automatically. * Revert "pgw#609/th#960 Phase 4: rewire CI to one suite, one lane" This reverts commit c39efb69b82d967393bd3f3243ce328d954380f7. * th#960/pgw#609 Phase 2b: absorb ~26 incident-pinned rows into P1/P2/P3/P5/ P6/P8/P9 + 2 new consolidated files, ahead of the bounded aggressive deletion sweep Bucket-level triage per the coordinator's ruling (Paul's directive: minimal greenfield set, ~92% removal). Absorbed only distinct bug-classes carrying a real pgw#/gw# incident number or a revert-catches-it story; untagged "tested thoroughly once" coverage goes without absorption (git history is the archive). P1 (+6): auth-rejection exit/retry-within-window, permanent-precondition fast-exit, hello-ack-deadline reconnect, not-leader redirect, hub-restart reconnect - absorbed from test_worker_grpc_e2e.py (#372) ahead of its deletion. P2 (+3): setup-failure FnUnavailable+recovery, host-RAM-failure wire ordering (th#807) from test_worker_grpc_e2e.py; corrupt-load quarantine+ retry-once (gw#408, J17 flood) from test_snapshot_corruption.py. P3 (+1): bare-ref prefetch binding resolution (#377) from test_executor_prefetch_binding.py. P5 (+1): heavy-dep stub contract (pgw#506) from test_discovery_heavy_deps.py. P6 (+1): SendQueue capacity-generation fencing from test_worker_grpc_e2e.py. P8 (+5): diffusers root-allinone skip (e2e J7 ENOSPC), standalone-component canonical-weight selection (gw#426), transformers sharded-index onnx/pickle exclusion, gguf explicit-quant refusal, non-safetensors-format refusal family - from test_classifier.py ahead of its deletion. P9 (+1): token-bound-owner routing (J19 run34) from test_media_upload_owner.py. New consolidated files (coordinator-authorized, one per lane with no P-test home): tests/test_quantization_lanes.py (W8A8/W4A4/FP8 contract + gw#409 promote-device-integrity, from test_w8a8.py/test_w4a4.py/ test_fp8_and_emergency_loading.py/test_promote_device_integrity.py) and tests/convert/test_convert_producer_contract.py (clone concurrency gw#442, disk preflight gw#462, download-skip bank th#592, publish resilience gw#462, from the tests/convert/ producer suite). Also reverted a premature local-only Phase 4 CI commit (c39efb6, from a prior session before an interruption) - never pushed, out of THIS issue's authorized scope ("do not touch CI yet"); preserved in history via revert, not discarded. Full new-suite gates: mypy clean, ruff clean, each new/changed file green in isolation. Deletion commit (the ~125 files these rows supersede) follows as a separate, deletions-only commit per the coordinator's mechanics. * th#960/pgw#609 Phase 3b: aggressive deletion sweep — 125 files (32,618 LoC) superseded or absorbed per the coordinator's ruling Every file here is either fully superseded by test_quantization_lanes.py / test_convert_producer_contract.py, absorbed as rows into P1/P2/P3/P5/P6/P8/ P9 (chaos b8274a1), or carries no incident pin (git history is the archive, per the coordinator's bound). Full manifest, bucket-by-bucket reasoning, and the dangling-import sweep are in the th#960/pgw#609 tracker Checkpoint 6, pushed before this commit. Deletions only, no other changes. Foreign-WIP clusters untouched: compile_cache/executor_adopt/fleet_cells (this issue's own claim-checkpoint note) and test_disk_telemetry_pgw610.py (landed on chaos by another lane during this session). Final state: 14 test files (3,293 LoC) + tests/harness/ (6 files) + conftest.py x2 + fake_hub.py, target was ~12-15 files per the coordinator. * pgw#610: lint — typed enum cast + drop unused import * pgw#610: CONTRACT.md — document StateDelta.disk_usage / DiskUsageReport * pgw#610: guard drain's disk-report cancel with getattr (stubbed Lifecycles skip __init__, same convention as _cancel_residency_reconcile) * gw#608 residual: AOT-cache disable must be process-global — torch 2.13 config user overrides are thread-local ContextVars Live disproof (0.40.5, B200, 2026-07-21): mint capture still packed 8 ASLR-keyed aotautograd entries and the store-served sibling still failed (warmups=1, cache_hits=0, cache_misses=8) — the 0.40.4 disable ran on the arming thread; the warmup compile thread saw the default True. The env var is read once at torch import (env_name_force), so post-import it is inert. Fix: entrypoint sets TORCHINDUCTOR_AUTOGRAD_CACHE=0 before any torch import (binds all threads + compile-worker subprocesses); _disable_aot_autograd_cache additionally mutates the installed config entry's env_value_force (process- global) for already-imported-torch embedders. Cross-thread revert-turns-red test added (fails on the 0.40.5 shape, passes with the fix). Dry evidence: fxgraph keys 8/8 identical across ALL five banked packs (0.40.1 x3 / 0.40.3 / 0.40.5); aotautograd level-2 keys 0/8 everywhere. samples/selfmint-proof/gw0405-run/. * th#964: family lane policy twin — bare conv-UNet (sd1/sd2/sdxl) bindings AUTO-pick scale-free #fp8 locally gen_worker.models.ladder gains family_root (modelfamily.Root twin) and the conv-UNet w8a8 exclusion table; a maybe_rebind_family_fp8 fold (hub-parity AUTO pick over the resolve's sibling_flavors, sm_89+ only, bf16 stays the sub-floor default; pins untouched) runs in the local CLI resolve ahead of the GGUF fold over one shared resolve. WorkerResolvedRepo carries the resolve's model_family (slot-declared family as fallback); select_gguf no longer lets an AUTO-ineligible w8a8 row suppress the GGUF pick. Shared vectors: none in this repo; no format change. * gw#613/th#965: universal app-level heartbeat — layer 2 of the 3-layer liveness contract (0.41.0) Hello declares heartbeat_interval_ms=30000; the pgw#610 disk-report task is promoted to the beat: it force-re-sends the full StateDelta every 30s from the asyncio event loop (the control loop that owes progress, never a detached thread), starts at startup() entry so boot hangs are covered, and keeps beating through drain until the stream closes. Edge-triggered deltas still flow as before; the beat is the same message re-sent unchanged. Contract §3 rewritten to the 3-layer model (lockstep tensorhub c03a531d). Tests: hello cadence, force-send bypasses edge suppression, beats flow while a startup coroutine is parked (the gw#612 shape), drain keeps beating until close. * gw#613 retune: 10s beat x 6 misses; disk stats measured every 30s riding every beat HEARTBEAT_INTERVAL_MS 30000 -> 10000 (hub reaps at 6 misses, ~60s). The pgw#610 statvfs/ref-index scan keeps its 30s measurement cadence behind a TTL cache so the 10s beat never turns it into a hot loop; the report still rides every StateDelta. Contract §3 event-loop discipline clause added — already true in 0.41.0: setup/warmup/compile/residency/GC all run off-loop (executor._to_thread_complete / asyncio.to_thread). * gw#612: gate self-mint publish on full capture coverage; finalize phase; run-26 root cause corrected (0.42.0) Run 26's filed post-seal_publish hang is disproven: the worker completed setup and advertised readiness; the wedge was the hub's singular compile fence treating the 2-lane record's same-identity self-attested targets as ambiguous (tensorhub lockstep fix). Real worker defect fixed: a shared mint capture missing an unexercised mandatory sibling's graphs was still published as the family cell, bricking every adopting boot (gw#611 qwen hits=1/misses=1). finalize_self_mint packs only; the executor publishes iff every capture sharer proved in, else withholds typed+loud. New 'finalize' activity phase covers the post-proof tail to readiness. * gw#613: soften run-26 narrative — app-silent, not proven-hung (gw#612 root cause: healthy-idle, hub fence starvation) * gw#613: marco_polo_wedge — event-loop-blocking liveness probe for th#965 layer-2 drills * gw#611: credit AOT-layer hits in the adopt warmup proof; calls= in the fail-closed detail; cross-process portability repro Measured mechanism for th#954's release-bricking 0/0: bundled AOT cache serves a fresh-process adopt with fxgraph counters fully silent, so a HEALTHY serving cell read cache_hits=0/cache_misses=0 and fail-closed. inductor_counters now surfaces aot_cache_hit/aot_cache_miss and the guard wrapper counts AOT hits as serving evidence (AOT stays pinned off per gw#608; a config regression now degrades to a proven boot, not a brick). Fail-closed detail carries calls= to split orphaned-wrapper (calls=0) from counter-blind serving (calls>0). New tests: real mint->pack->fresh- subprocess adopt asserts >=1 FX hit; bundled-AOT 0/0 mechanism pin; guard credits AOT hits (red on revert). Hub lockstep: tensorhub chaos 9e7dca8e quarantines cells failing their own adopt proof. * th#767: gen_worker.families.wan — WanDefaults (animegen lane) Second family after SdxlDefaults: steps/guidance/guidance_2/max_guidance for the Wan 2.2 A14B MoE envelope. Registered under tensorhub's canonical architecture root "wan22" (not a Compile(family=) shape-grid string) so repo-metadata PUT validates against the matching tensorhub schema (companion wan22.schema.json on tensorhub chaos). Lets the wan-2.2 endpoint's T2V function migrate to the Slot model and give AnimeGen-T2V its own distilled recipe (8 steps, guidance=1.0, guidance_2=1.0) via repo metadata instead of branching on checkpoint identity. * gw#614: synthesized media-modality warmup coverage (union family cells publish) + on_hello_ack model-set-diff cancel (A) _run_synthesized_warmup coverage pass: compile-target objects still unexercised after the planned jobs get media VARIANTS of the same base payloads (warmup.media_variants: one optional image/audio field filled with a generated asset, nothing else drifts) — input-routed sibling lanes (qwen edit) mint into the shared capture, so the gw#612 gate publishes the union cell instead of withholding forever; adopts prove the sibling lane instead of arming unproven. (B) th#961 defense in depth: a HelloAck whose semantic model set (resolutions+disk_refs+snapshots+hot) matches the running reconcile's target no longer cancels it (in-flight self_mint_compile survives benign plan rewrites); changed sets cancel as before. * gw#615: fix event-loop-blocking disk telemetry — the 0.40.7 LTX post-seal_publish hang pgw#610/th#962's disk_usage_report() ran statvfs()/stat() synchronously inline in Lifecycle._state_delta(), which many hot paths call directly with no event-loop offload — including right after a self-mint's seal_publish. The provider-attached VOLUME fill-source is a network-backed mount that can stall for minutes under load (exactly what a self-mint's weight download + cell pack produce). A stalled statvfs() there freezes the whole event loop: every StateDelta, RunJob dispatch, and even the th#965 heartbeat (same loop) stops until the syscall returns. ModelStore.disk_usage_report() now only reads a cache; the actual measurement (_measure_disk_usage_report) runs exclusively via refresh_disk_usage_report(), off-loop through asyncio.to_thread. Lifecycle kicks that refresh as a fire-and-forget background task (never awaited inline), gated to the existing ~30s TTL, so a stalled mount delays only its own telemetry freshness — never any send, dispatch, or heartbeat. Root-caused via git diff v0.40.5..v0.40.7 (executor.py/lifecycle.py) plus gw#612's hub-log forensics for the qwen case: gw#612 itself was hub-side (dispatch-fence starve, corrected root cause, no worker hang) and its worker-side companion fix addresses a different defect (poisoned family-cell publish on the NEXT boot). This is a distinct, genuinely-new gen-worker regression introduced by a70e7c9 (wave-2 telemetry), reproduced locally with no GPU via a stalled-statvfs monkeypatch. Tests: tests/test_disk_telemetry_pgw610.py (new test_disk_usage_report_never_measures_only_reads_the_cache, test_state_delta_never_blocks_event_loop_on_a_stalled_mount; existing tests updated to the cache/refresh split) + tests/test_heartbeat_gw613.py (new test_beats_flow_while_disk_measurement_is_stalled; existing TTL test updated to the fire-and-forget refresh). All revert-turns-red verified. Full suite 458 passed/5 skipped; ruff+mypy clean. * gw#608: FX-key forensics — a store-served proof failure names the diverging FxGraphHashDetails component; compile_seconds + per-object proof counts in the fail-closed detail * gw#608: clamp fx-divergence detail under the 2000-char activity error cap * gw#608: exhaustive fx_cache_failure_report — counts + same-key re-save sibling diff + load probes discriminate B1/B2 in one boot * gw#608 FIX: self-mint arm is transactional over the process-global cache env — no-target siblings decline before any dir mutation, arm failure restores the prior env, and a delivered-cell-seeded process never opens a capture (the store-served 8/8-miss root cause) * gw#585: consume tensorhub v4 private-input manifests — v3->v4 hard cut (PR #320 port onto chaos) Proto adopted byte-for-byte from tensorhub chaos 07e5520b (protocol 4, RunJob.input_assets=15 — 14 taken by th#913 lane — InputAsset fields 1-6; CLI NDJSON stays v1); pb2 regenerated with grpcio-tools 1.82.1. Worker resolves the exact ordered manifest via one bounded resolve call under the attempt-scoped capability, verifies size/BLAKE3/MIME/kind/decode, preserves opaque refs, sets only local_path, attempt-owned temp cleanup; endpoint build rejects Asset-bearing set/frozenset and non-string maps; deterministic traversal (declaration order, list index, sorted string map keys). Full local suite 558 passed; mypy/ruff/ timeout-lint clean. uv.lock: mechanical 0.42.0 catch-up to pyproject. * gw#585: marco-polo-attach — private-input echo probe (blake3/size/local_path, optional in-flight delay) for the v4 e2e lane * gw#585: GEN_WORKER_INTERNAL_OBJECT_HOSTS — exact-host allowlist for resolver-minted private-input URLs (datacenter-internal object stores); caller public transports keep the full SSRF gate * gw#585: marco-polo-attach reads via os.fspath (Asset is os.PathLike; no .path() method) * gw#608: fix publish-event race in the self-mint adopt test fake — publish() fills the published dict before signaling the event (CI KeyError: 'bytes') * pgw#617: hierarchical slot bindings — load base composition, substitute components from ModelBinding.components - proto: ModelBinding.components (field 5, map component -> canonical ref; mirrors tensorhub th#980); pb regenerated - ModelRef.component_overrides (tensorhub-only, sorted pairs) — part of the binding identity, so a component-only rebind derives a new instance/ residency identity; flat bindings byte-identical - executor: _effective_spec stamps overrides per dispatch; setup materializes each override ref from its own snapshot and injects the loaded module via the gw#479 components= from_pretrained mechanism; pins/held refs/digests/compile-cell binding facts include override refs - loading.load_component_override: class from the BASE model_index.json, weights from the override tree's subtree/root - typed ComponentSubstitutionError (unknown component, non-CAS ref, non-worker-loaded slot) - registry: selected_by= slots may omit default_checkpoint (th#980 relaxation mirror; unblocks ie#524 sweep of wan-2.2/sdxl/z-image) - tests: hub-double flat->substitute->flat identity proof, typed unknown- component refusal, registry relaxation (564 green) * pgw#617: mypy — dedicated slot_comps var in _effective_spec * gw#619: report a typed hardware-unsuitable fact to the hub before the boot-time CUDA probe's silent exit entrypoint.py's boot sequence (gw#529 cuda_probe) logged GEN_WORKER_CUDA_PROBE_FAILED and exit(1)'d with zero orchestrator contact — invisible to every layer of telemetry (th#986: ~20 pod deaths whose cause, CUDA-13 images on old-driver hosts per th#988, was unattributable). On a probe failure the worker now dials the hub and sends ONE HardwareUnsuitable WorkerMessage in place of Hello (cozy.scheduler wire addition, tensorhub th#988 companion commit) before exiting — bounded best-effort (~7s worst case: 2 attempts x 3s + 1s backoff), never a reason to delay the exit materially, and a silent fallback when the hub is unreachable or predates this field (old-hub FAILED_PRECONDITION rejection). cuda_probe.py: classify_probe_failure() — typed reason_class vocabulary (torch_unavailable | cuda_unavailable | driver_too_old | cuda_error | unknown) derived from CudaProbeResult.reason; driver_too_old matches torch's own "driver too old (found version ...)" message verbatim (the exact th#591/th#979/th#988 signature). hardware_report.py (new): builds the report (nvidia-smi/NVML for driver_version+gpu_name — reads the driver directly, independent of the broken CUDA runtime being reported on; torch_version/torch_cuda_version; gen_worker_version; image_digest/instance_id from Settings; worker_id/ release_id from Settings or JWT claims) and dials it over a one-shot grpc.aio Connect stream. entrypoint.py: calls report_hardware_unsuitable() right where the probe failure was previously silently fatal; logs a cuda_probe_hardware_report startup phase either way. Tests: tests/test_hardware_report.py (classify_probe_failure vocabulary; build_hardware_report degrades safely without nvidia-smi and reads driver identity when torch.cuda is down; report_hardware_unsuitable delivers to a real hub-double socket, resolves identity from JWT claims, and returns promptly-false against both an old-hub protocol rejection and a connection-refused unreachable hub). tests/test_boot_smoke_hardware_report.py (true end-to-end: a real `python -m gen_worker.entrypoint` subprocess whose probe fails on THIS box's actual driver/torch mismatch dials a real hub socket, and the hub-unreachable fallback still exits without hanging). tests/harness/hardware_report_hub.py: shared real-socket test servicers (recording_hub, old_hub, closed_port_addr). Full suite green (578 passed) plus the extended gw#591 boot-smoke contract (tests/test_boot_smoke_gw591.py, tests/test_p7_boot_smoke.py) confirmed unaffected. * ie#522: non-inference functions must never report loading_functions() _warmup_plan already excludes non-"inference" kinds from boot warmup (spec.kind != "inference"), but available_functions()/loading_functions() didn't share that gate: a declared-but-never-dispatched conversion function (a release can bundle many; only a few get invoked) sat in loading_functions() forever, since nothing was ever going to warm it. The hub's th#965 layer-3 stall watchdog takes loading_functions() at face value and kills a worker after 10 minutes of "no open activity" on any name in that list — including a name nothing was ever loading. Reproduced live: a wan-2.2 conversion release bundling z-image-w8a8-quantization alongside clone-huggingface got its pod killed at 10m15s three times running, on three different hosts, each time mid a legitimate multi-GB clone-huggingface transfer. Fix: conversion/training/dataset-kind functions are always available (matching the existing spec.slots per-dispatch-serveability carve-out just above) and never counted as loading. * ie#522: watchdog evidence blind to I/O-bound download+load phases The activity watchdog's default liveness evidence was process CPU seconds only. Two real activities are honestly alive while burning near-zero CPU: an I/O-bound network model fill (large composites, tens of GB, CPU-light by design), and the subsequent on-disk load into GPU (safetensors mmap + device placement — third-party diffusers/torch code with no progress hook). Both read as "stalled" to the hub's th#965 layer-3 rule after 10 minutes of no heartbeat, even mid genuine progress. Reproduced live twice: identical ~9.5min self_mint_compile CancelledError at phase=load, two different RunPod hosts. Fixed three ways: - _process_cpu_evidence now sums live (not just reaped) child-process CPU via psutil — covers inductor's async_compile subprocess workers, which resource.RUSAGE_CHILDREN can't see while they're still running. - note_progress() lets an I/O callback (model-download byte ticks) heartbeat the current activity directly, independent of the CPU-sampling watchdog thread; wired into cozy_snapshot.py's _on_bytes network-byte path. - _process_io_evidence adds process disk I/O bytes (psutil io_counters) to the default combined evidence — a universal, non-invasive signal for the on-disk load phase that needs no app-level hook into third-party loading code at all. Revert-turns-red verified: reverting activity.py/cozy_snapshot.py makes exactly the 3 new evidence tests fail (zero-CPU download, zero-CPU disk load, live-child-CPU), while the pre-existing dead- activity-still-stalls test keeps the detector honest. Full suite green (584 passed, 5 skipped, 1 xfailed). * gw#621: progress registry — named monotonic counters on the 10s beat + self-diagnosis New gen_worker.progress: phases register named monotonic counters (download:<ref> bytes w/ snapshot-known totals, evidence:<kind> riding the watchdog, warmup:jobs, infer:steps, upload:bytes); family prefix maps to a per-phase self-diagnosis window (code constants). activity.on_beat rides lifecycle's 10s heartbeat: one counter-carrying ActivityUpdate per beat — frozen counters included (the hub kills on counter non-advancement) — and self_stalled=true + stalled_for_ms when even the freshest counter is past its window (kill on confession, never CPU inference). Additive ActivityUpdate fields 10-16 (counter, unit, done, total, rate, self_stalled, stalled_for_ms; th#994 carries the proto source). Counters are activity- owned (finished at activity end). Downloads now carry bytes_total on the FIRST DOWNLOADING event when the snapshot knows sizes — no more 'started total=0' then silence. Also completes ie#522's 7640551, which swept the in-flight activity.py half of this work into its commit — gen_worker.progress ships here. * gw#621: tests — registry windows/self-diagnosis, counter beats, hub-double e2e confession Revert-turns-red for the ie#522 false-positive class: a CPU-quiet byte-advancing download is visibly alive on the wire (counter beats, never self_stalled) over a REAL Worker + hub double; a frozen counter still confesses, and local behavior is unchanged (the hub owns the kill). * gw#621: progress_beat re-checks _done — a beat racing completion never re-opens a terminal activity hub-side * ie#522: arm_compile() no-ops (never raises) with no active arming scope Endpoint setup() declares intent by calling arm_compile() unconditionally (sdxl, wan-2.2, ...); whether that intent is active is the release's decision — an eager registration (no compile=Compile(...) declared) already opens ArmingScope(None, ...) as a documented no-op, but arm_compile() itself still raised RuntimeError when called with no active scope. Reproduced live: wan-2.2's eager smoke (dropped compile= to skip a 51.5min self-mint compile grid per Paul's ruling) crashed at warmup/phase=load on this exact raise. Fix: no active scope -> log at info, return False (never armed), same as every other "stays eager" path. Revert-turns-red verified (4 tests updated/added in test_arm_compile_pgw517.py). Full suite green (594 passed, 5 skipped, 1 xfailed). * pgw(th#1004): @variant_of handler annotation — variant pairing emitted into the discovery manifest (variant_of/variant), dangling/self/chain targets fail the build * ie#522: WanDefaults gains an optional shift field (scheduler flow-match shift) Live finding #3: the AnimeGen card's own 8-step recipe pins FlowMatchEulerDiscreteScheduler(shift=3.0) alongside its distillation LoRA -- a step-distilled lineage needs a non-default scheduler shift, which nothing in the wan22 inference-defaults vocabulary carried. shift: Optional[float] = None -- every non-distilled repo's metadata and the resolved-defaults fallback leave the pipeline's own scheduler config untouched; only a repo that explicitly publishes shift gets one applied at the endpoint's resolution site (wan-2.2 main.py, separate commit). Mirrors tensorhub-chaos's wan22.schema.json shift addition. * ie#522 fix: de-flake the zero-cpu watchdog test for constrained CI runners test_watchdog_survives_zero_cpu_download_via_note_progress was flaky by construction: in its 0.3s real-clock window the 5s note_progress rate-limit floor allowed exactly ONE beat, so the >=2 assertion silently depended on incidental watchdog CPU/IO evidence noise for the second beat — present on a 32-core dev box, absent on 2-core GitHub runners (confirmed by per-thread beat attribution: ticker contributes 1, the rest come from the watchdog thread; beats=1 reproduces even locally outside pytest). Deterministic rewrite, intent intact (zero-CPU download must not go silent while note_progress ticks arrive): drop the rate-limit floor for this test (the floor is covered by its own test), pin watchdog evidence to a constant zero so every counted beat is attributable to note_progress alone (airtight revert-turns-red — verified: neutering note_progress fails it with 0 beats), and wait event-driven on beat count with a 10s deadline instead of a fixed wall-clock window. Verified 20x unconstrained, 10x taskset -c 0, 5x taskset -c 0,1; full file green; mypy clean. * pgw#622: eager-while-compiling with hot-swap — novel request shapes serve immediately A novel input signature at a compiled target no longer stalls 30-60s behind Dynamo+Inductor: the guard routes it to the eager original, one background thread warms the compiled callable with a zero-filled dummy batch (separate CUDA stream, nice +10), and a successful warm atomically hot-swaps the signature to the compiled path. Each completed warm repacks the live cache root and republishes the cell under the same key so the fleet never compiles that (shape, GPU, lane) again. Sequential compile-then-serve is kept for: the boot warmup-proof window, mandatory quantized lanes (w8a8/w4a4), tight VRAM headroom (degrade, never OOM), regional mode, and signature-vocabulary explosions. * release: gen-worker 0.45.0 — pgw#622 eager-while-compiling with hot-swap * release: 0.45.0 uv.lock version sync * th#1017: SdxlScheduler gains lcm/euler_trailing distilled-regime schedulers * th#1017: @endpoint regimes= decorator surface + manifest emission Per-method {name: (regime,...)} dict on class handlers (mirrors warmup=), bare tuple on the function form, default ("standard",). Validated against {standard,v_prediction,distilled}; EndpointSpec.regimes carries the resolved per-handler tuple; discover.py emits "regimes" only when it differs from the default (absent = ["standard"]). * th#1017: ResolvedSlot.regime + executor-level regime backstop resolve_slot/resolve_slots thread inference_regime (default "standard") into ResolvedSlot.regime, sourced from the hub resolve response's inference_regime field (hub_client.WorkerResolvedRepo, mirrors model_family; "" on hubs not sending it). RegimeMismatchError raises when a resolved checkpoint's regime is outside the invoked function's declared regimes= — wired as a backstop in executor._resolve_slots_kwargs via allowed_regimes=spec.regimes. * th#1017: converter stamps regime-correct scheduler config run_clone/build_flavor_tree accept an inference_regime hint (payload field of the same name); writer.apply_regime_scheduler_config rewrites a produced diffusers flavor's scheduler/config.json: v_prediction sets prediction_type + rescale_betas_zero_snr, distilled sets trailing timestep_spacing. No-op for standard or a tree with no scheduler component. Updated the P8 build_flavor_tree test stub to accept the new kwarg. * th#1017: tests — regimes decorator surface, manifest, resolution, converter Real discovery/registry/slot/convert codepaths (no mocking): decorator validation errors, class/function manifest round-trip + default-omitted, ResolvedSlot.regime + RegimeMismatchError backstop, hub_client inference_regime parsing, and scheduler-config stamping for all three regimes. * gw#623: only cancel an in-flight residency load when the ACTIVE model leaves the set Root cause of the ~9.5min (and ~15s-burst) warmup/self_mint_compile CancelledError at phase=load on nearly every serving boot: _semantic_model_key hashed full snapshot serializations INCLUDING presigned file URLs, which the hub re-signs on every ~15s release-config rebuild and deliberately excludes from its own HelloAck semantic hash. Every resent ack carrying rotated URLs (or any sibling-ref churn, e.g. the th#1021 preposition/seed fight) looked like a model-set change and killed the in-flight load. - _snapshot_content_key: digest + (path, size, blake3) per file; URL excluded - level-triggered reconcile: passes re-read the freshest desired set; a benign mid-load update signals a re-pass instead of cancelling - cancel only when the actively-loading item left the set or changed snapshot identity / precision resolution (_active_work_still_desired) Tests (revert-turns-red verified): URL rotation keeps the load, sibling ref added mid-load keeps the load then converges, active ref leaving cancels, active-ref resolution repick cancels. * gw#624: free cancelled load/warmup attempts' memory before the retry Live: 5 cancelled load retries on one worker climbed container RAM 3%->97% and VRAM to 83.86GB (OOM on an 80GB card) — each cancelled attempt's partially loaded modules stayed alive (pinned by the CancelledError's traceback and by uncollected reference cycles) while the retry loaded a fresh copy on top. - _to_thread_complete: drop the joined Task reference on the cancel path so the discarded load result is not pinned by the traceback frame during rollback - _injection_kwargs: release shared-component holds acquired before the failure (they never reached rec.shared_keys, so vacate could not) - _rollback_failed_setup: gc.collect + cuda.empty_cache after vacate, and schedule an allocation purge - _setup_locked: run the purge (gc + empty_cache) under the load lock BEFORE the next attempt allocates — a retry provably starts from baseline Tests (revert-turns-red verified): cancelled _to_thread_complete result is unreachable while the exception lives; cancelled ensure_setup's buffers are gone at the next attempt's setup entry (gc disabled for the duration). * release: gen-worker 0.46.0 — th#1017 inference regimes * th#1031: cell_selection_bug recovers via self-mint instead of retry-blocking forever A self-requested cell whose graph signature drifts from this runtime's own (cell_key has no graph-shape axis, so structurally different sdxl graphs can share one key) used to raise CellSelectionBugError straight out of fleet_cells.enable_compiled, aborting the whole arm attempt. For a mandatory w8a8/w4a4 lane that becomes CompiledLaneUnavailableError, which fails setup entirely and gets retried from scratch on the next request/job -- hitting the identical stale delivered cell again, forever. Live pod_events proves this: 54 cell_selection_bug rows across 2 releases/9 pods over 24h+ (predating pgw#622 by ~12h, so unrelated to it), all rejecting one stale signature (c3c4395265ad) that nothing was ever able to overwrite. ArmOutcome now carries the caught CellSelectionBugError; enable_compiled falls through to self-mint (the ordinary MISS recovery) instead of raising. The th#883 invariant stays wire-visible -- callers still send the cell_selection_bug ModelEvent/pod_event -- it just no longer blocks serving. Self-mint's own fail-closed safety net (no CUDA/toolchain/target) is untouched, so a genuine mint impossibility on a mandatory lane still refuses, now chained from the selection bug via __cause__. Threaded through both compile-arming seams: the worker-loaded slot path (_enable_compiled) and the self-loaded pipeline path (ArmingScope/ arm_compile, what sdxl actually uses) via a new selection_bugs map so neither silently drops the alarm. * th#1043: joint precision fit for shared-component multi-lane loads Sequential per-lane loading let the first lane in a gw#479 shared- component group (qwen-image's t2i/edit) consume free VRAM at native precision, starving the second lane into an offload placement the shared-component invariant refuses outright ("shared-component lanes require resident placement"). Decide fp8 storage for the whole group up front, against its combined footprint, before any lane loads. * th#1043: strict_vram refuses a load-time CUDA OOM instead of offloading place_pipeline's reactive OOM ladder had no strict_vram awareness: a CUDA OOM during placement would silently walk into model_offload even for a binding whose Resources(strict_vram=True) declares it cannot tolerate CPU-resident weights. Plan-time already refuses a PREDICTED offload under strict_vram, but a load-time OOM the plan didn't predict (e.g. a deploy-time binding that no longer matches the author's vram_gb sizing) reached the offload ladder anyway. Complements e4f9f6f's joint precision fit — defense in depth for any strict_vram binding, not just qwen-image's shared-component case. * th#1043: forced group-fit fp8 survives the bf16-resident upcast (0.48.1) (#375) Live 0.48.0 pod showed the joint fp8 forcing being silently un-done by load_from_pretrained's single-lane resident-upcast check (gw#534 rung 2): the first lane fit current free VRAM alone, upgraded back to bf16, and re-starved its sibling into the refused offload placement. A forced storage dtype now disables the local upgrade — the headroom belongs to the group, not the first lane to load. * th#1043: forced group-fit fp8 reports as a structural degradation (0.48.2) (#376) The joint shared-lane forcing served fp8 storage while the hub's plan said native bf16 — no FnDegraded, so placement never learns the card is too small for the group at native precision. Stamp the same rung outcome as the adaptive ladder. * th#1050: SDK lane contract — ctx.lane + opt-in handles= lane declaration (0.49.0) - ctx.lane: read-only executing lane on RequestContext, set post-degrade from the same truth as JobMetrics.lane; boot-warmup + CLI runs covered - @endpoint(handles=[...]): opt-in declared lane bodies (divergence marker, never inventory); validated against known concrete bodies; emitted into the discovery manifest as fn.handles - lane instruction: a declared lane is satisfiable with no laddered rebind (author code branches on ctx.lane); undeclared lanes refuse typed as before - hub-double e2e test: declared endpoint branches on ctx.lane over the real gRPC wire; JobMetrics.lane matches; undeclared dispatch unchanged * th#1051: RuntimeFormula SDK surface + worker-side term evaluation on the back-channel (0.50.0) * th#1051: uv.lock version bump * th#1051: lock gen-worker 0.50.0 * th#1051: mypy-clean Constant handling in formula eval/canonical * th#1055: desired-hot warm works on slot-only endpoints; refusals emit model events (0.50.1) ensure_desired_instance required the hot instance's binding set to equal spec.models, but deploy-bound Slots (ie#524/th#980) have no code default so spec.models is empty fleet-wide — every hub hot intent (gw#587 self-mint prewarm, th#912 seeding, #567 cell reload) failed a ValidationError that _reconcile_pass swallowed as one pod-local warning: no warmup, no self-mint, the th#868 w8a8 fence never opened, cells never armed. - validate against the declared slot set; code defaults may fill their own - declared-space hot bindings remap through the HelloAck precision picks - every desired-instance failure emits MODEL_STATE_FAILED for the instance refs (the silent stall is now impossible) * th#1055: version 0.50.2 (0.50.1 taken by pgw#626 on master) * th#1063: loud boot log when a datacenter pod has no CAS fill source (0.50.3) A RunPod pod without TENSORHUB_FILL_SOURCE_DIR (or with a non-mount path) pulls everything from R2 with write-through silently off — the defect that hid the fleet-wide unattached-volume regression. Boot now logs fill_source_disabled (reason=unset|not_a_mount) on datacenter pods and fill_source_enabled when the volume tier is live. * gw#627: Conv2d additive-branch support in the runtime LoRA overlay (0.51.0) Curated sdxl distill adapters (Lightning/DMD2) carry 49 conv LoRA pairs; the gw#547/558 branch was Linear-only, leaving raw peft as the only route on the w8a8 lane — fatal on _Fp8ScaledLinear (8/8 live turbo failures, th#1037 addendum). Plain nn.Conv2d denoiser modules are now branch-capable: canonical zeroed conv branches in the same rank bucket, staged-copy swap, eager instance-forward wrap conv1x1(conv(x,A),B). * pgw#628: residency reporting v2 — content-addressed idempotent observations (0.52.0) Worker half of th#1070: every applied HelloAck opens a republish epoch; the reconcile pass re-announces verified cached identities (exact ref+digest) once per epoch even when unchanged — a re-received plan (redrive/overdue resend/reconnect) is the hub asking for a resync, and success observations are safe to emit late, twice, or across plan revisions under the v2 hub's digest acceptance. Job-path cache hits within an epoch stay deduped. gw#614 same-set no-cancel unchanged. Revert-turns-red test; mypy+ruff clean; full suite 649 passed/5 skipped. No forced fleet rebuild — v2 hubs accept 0.44-0.51 reports fine. * gw#627 live fix: enable_compiled skips the branch lane on slot objects with no compile target A bare component slot (sdxl's AutoencoderKL vae) resolves none of cfg.targets; apply_lora_lane's no-denoiser error broke the whole model load (release-broken streak on the first gw627 deploy). Lane applies only to objects that resolve a compile target; the loud misconfig error stays for real targets. * gw#627 live fix 2: reject legacy attn-processor output from lora_state_dict normalization diffusers' non-diffusers converter turns kohya-flat sdxl attention keys into legacy …attn1.processor.to_q_lora.down.weight names that match no real module — the whole curated adapter failed typed (1680 unresolved keys, live request 88604466). Fall back to the raw keys, which resolve directly against module paths. gw627 test now runs the REAL SDXL converter in the loop. * th#1084 (pgw side): input-caused refusals raise ValidationError -> INVALID (0.53.0) RepoRefusal + SourceIncludeError are ValidationErrors; HF repo-access errors (RepositoryNotFound/Gated/Revision/EntryNotFound) wrap as typed input refusals at the convert ingest boundary with the class name preserved; an all-input-rejection 'no publishable flavor' aggregate is INVALID. Bad user repos fail only their request — never the release. * gw#631: delivered lora-bucket cell at a bare component slot is an ordinary miss, not a CellSelectionBugError r3 live chain (pod xzhlioakq909wq): the self-key check computed the family lora64 key with cfg.lora_bucket even when no cfg.target resolves on the slot object (sdxl's standalone vae), claimed the delivered cell as its own, and raised on lane drift (weight_lane 'lora64' != pipeline '') -> gw#608 seeded-cell self-mint refusal -> cell quarantine -> pod retired all_declared_functions_disabled. 0.52.1 downgraded apply_lora_lane for targetless slots; this applies the same effective-bucket rule to the self-key. * gw#632: renumber comments (id raced with slim-down filing) * th#1087 stages B+D: declared config params/envs (ConfigParam, env=), ctx.config, gen-push reconcile + local snapshot file for subprocesses (0.54.0) * th#1087: align worker config reconcile with final protocol * gw#631: remove unused worker and SDK surface * fix(config): pin subprocess snapshots per invocation * gw#631: retire unused H2D canary field * gw#496: remove silently dropped checkpoint metadata * gw#496: trim unused dataset conveniences * Project lifecycle shadow state from workers * Centralize worker lifecycle intent reporting * feat: report worker lifecycle intents * fix: restore RunJob org ownership contract * Implement exact config convergence protocol * marco-polo: regenerate uv.lock for editable gen-worker 0.56.0 * th#1098: marco-polo example 1.0.1 — gen-worker range to >=0.56,<0.57 (protocol v5), relock * pgw#636/#637/#638: hot-GPU mandate — pack VRAM with checkpoints, credit dynamo's in-memory cache, stop starving staged downloads pgw#637 (P0, the 2026-07-24 sdxl root cause). Cell keys are checkpoint-free, so the 2nd checkpoint of an already-proven family serves its warmup from dynamo's in-memory compiled code with ZERO FX/AOT counter movement (calls=1, hits=0, misses=0) and was deterministically judged "disproven" -> CompiledLaneUnavailableError -> FnUnavailable compile_cell_failed on every worker. The finalize proof now credits that signature, gated on BOTH: 1. compile_cache.cell_proven_in_process(ref) — this process actually served that exact cell (a real FX/AOT hit, or a finalized self-mint), and 2. compile_cache.has_inmemory_compiled_code(pipe) — DIRECT evidence from torch._dynamo that compiled code is live for THIS object's targets. Both are load-bearing: (1) alone lets a sibling object's hit certify this object's silence, which gw#603/gw#611 forbid (and which broke two guard tests); (2) alone would credit a cell never proven anywhere. Aggravator fixed: apply() tracks armed pipelines in a WeakSet and unwrap() skips the PROCESS-GLOBAL torch._dynamo.reset() while a sibling is still armed, so a failing 2nd arm no longer kills the healthy 1st checkpoint's lane. pgw#636 (Paul: "vram_gb means 'at least X', not 'only X'"). The one-pipeline ceiling was the pre-load ask: max(sum(vram_hints), Resources.vram_gb) reserved 12 GiB (+2 GiB margin) for every never-seen checkpoint pick and evicted the resident pipeline on every hop. _estimate_setup_need now prefers a prior MEASURED vram_hint, else the wire snapshot's real byte total, and falls back to the declaration only when a ref has no byte facts at all. Slot( share_components=...) opts a dynamic slot into cross-pick content-keyed component sharing (gw#479 generalized, content-honest: unequal bytes never alias). Residency shared entries move from refcount to `holders`: referenced components are demotable while idle (owners pin+promote before executing), never evictable while referenced, and 2+-holder entries sort LAST in the LRU; record vacate no longer eagerly drains them. pgw#638. Disk staging is network+disk I/O and holds no GPU lock, so _reconcile_pass no longer gates it on executor.wait_idle() and on_message(run_job) no longer cancels it (only an in-flight HOT warm is preempted). Live: staged 4.7 GB downloads sat at 0% for 4+ minutes on busy workers while the hub bought pods for models the fleet nearly had. The reconcile loop parks on wait_idle() OR restart after a pass on a busy worker, so a fresh HelloAck is never stuck behind a long tenant job. Tests (real code paths; fakes only at the torch/network/capacity boundary): - test_compile_proof_pgw637.py — proven-cell registry, the dynamo probe (negative and positive), unwrap sibling-scoping. - test_executor_adopt.py::test_second_checkpoint_served_from_dynamo_inmemory_cache_proves — the filed regression through the REAL ensure_setup finalize path, plus the same counter signature WITHOUT live dynamo code still failing closed. - test_hot_gpu_pgw636.py — real _estimate_setup_need, real Residency holders semantics and LRU ordering, the real make_room packing juggle (four ~5 GB picks hot on a 24 GiB budget, one LRU victim under real pressure, warm promote back), serve-while-downloading over the real worker + hub-double, and the end-to-end juggle: four checkpoints through ONE selected_by= slot, round two runs zero additional setups (all four instances retained). * pgw#639 (partial): SIGUSR2 -> faulthandler stack dump of every thread The worker's asyncio loop owns transport and the 10s heartbeat while all model work runs on threads, so a thread wedged inside a C extension keeps heartbeating and hub-side logs can never see it. Registering faulthandler on SIGUSR2 at boot makes 'kill -USR2 <pid>' print all thread stacks to stderr, allocation-free, so it also works under memory pressure. This is the filed 'trivial first step'. The delivery half (getting a signal into the pod: sshd behind an env flag, or a hub-mediated debug RPC) is still open on pgw#639. * th#1107: three serve-path fixes — vae_only stops taxing pipelines that fit, strict_vram binds the auto SELECTION, image path signals the finalize handoff vae_only is chosen when the pipeline FITS and only headroom is tight, yet it enabled VAE tiling + attention slicing (memory.py:672) — tiled decode re-runs the VAE per tile and attention slicing replaces the fused SDPA/flash kernel. Both are now reserved for the rungs where the model genuinely does not fit. strict_vram was enforced only inside the OOM-demotion branch, so select_auto_mode returning model_offload/group_offload placed CPU-resident weights with no OOM and no error — the th#1043 shape reaching serving as a silent 5-10x tax. write_image never called _release_gpu_slot_for_finalize, so gw#516's overlap was dead on every image endpoint: finalize_wall_ms == 0 on 19 of 20 endpoints in the master hub DB (979 sdxl samples), non-zero only on ltx-video (median 3.25s of a 14.9s request). The encode+upload tail now runs slotless like the video path. * gw#640/th#1077: worker fatals reach the hub, not just pod stdout entrypoint._log_worker_fatal wrote the exception class, message and traceback to the pod's stdout ONLY. RunPod exposes no container-logs API, so a cloud-only worker death was un-debuggable by construction — the th#1085 cold-boot investigation burned six live runs on a crash whose traceback existed and was unreachable. The fatal is now also dialed to the orchestrator over a fresh Connect stream, reusing the HardwareUnsuitable carrier with reason_class="worker_fatal". The hub already persists that message as a durable pod_events row (class + reason + full JSON payload), so this needs no proto change and no hub redeploy, and works against every hub pin already deployed. Second half: the run loop ending WITHOUT a hub Drain or a shutdown signal was a clean, silent `return 0` — exactly the gw#640 signature the hub could only observe as a young-worker death. That is now a reported UnexpectedWorkerExit fatal. Worker.run() keeps its int contract and reports from the sync wrapper; the cuda_probe path is left alone because it already sends its own typed HardwareUnsuitable report. * pgw#638: revert serve-while-downloading — the naive fix breaks the mutable-tag identity fence Dropping the tenant-idle gate on disk staging (and the run_job cancellation of the reconcile pass) is the obvious fix for the measured starvation, and it is wrong. The pass's first act for a ref is to fence a stale resident identity via revalidate_snapshot_identity; a tenant job may legitimately be re-materializing OLDER bytes for that same ref concurrently. Unserialized, the fence loses the race and the worker reports the stale identity indefinitely — the th#1066 hub/worker residency drift class. Evidence: test_p2_residency_reconcile.py:: test_mutable_tag_move_fences_events_by_digest_and_generation goes red and does not converge within 10s of polling. Two attempted patches also failed: - re-verify after every job — handle_run_job returns BEFORE the job's load completes, so the re-verify still races it; - gate only when a fence is actually pending (needs_identity_fence) — the predicate is evaluated before the job's record becomes ready, so it reads False and skips the gate exactly when it is needed. The correct fix is pgw#641 Stage 4: transfers become their own tasks with their own lifecycle, so staging never sits inside this serialized pass and never has to choose between progress and ordering. Recorded in pgw#638. pgw#636 and pgw#637 are unaffected and stay. Also drops the `hold` toy endpoint and the serve-while-downloading test that only existed for the reverted behavior; the four-checkpoint juggle test stays. * pgw#650: legacy sweep — delete the pre-slot ModelChoice binding vocabulary, the dead ar_tts runtime, and 12 zero-consumer symbols The model SET is catalog, not code (th#980/ie#524/th#1087): Model(id, Hub(...), defaults) rows embedded a curated checkpoint list in endpoint code, superseded by Slot(selected_by=) + hub mapping. Zero endpoints declare one; the model.choices[] manifest block has never been emitted in production. - delete api/model.py (Model/ModelChoice/ModelDefaults/is_model_choice) and discovery's _model_choice_in/_collect_model_placement_key + fn["model"] block - delete runtimes/ar_tts.py: zero importers, zero tests, zero docs, zero consumers; runtimes/__init__ advertised it in __all__ without importing it - delete 12 symbols with no reference in src, tests, or any consumer repo: resolve_slots, DestinationRepo, read_gguf_metadata, materialize_safetensors_input, is_calibration_required, is_inline_supported, spec_id, snapshot_weight_bytes, with_oom_retry, quantized_modules, content_credentials._reset_for_tests - tests: delete test_request_context_checkpoint_contract.py (inspect.signature only, executes no behavior), test_boot_smoke_gw591.py (assertion-identical duplicate of test_p7_boot_smoke.py, pays a second real torch subprocess boot per run), test_families_wan.py (WanDefaults has no consumer; resolve_slot precedence is covered by P3), test_testing_helpers.py (tests the test helper); drop two rows that assert a protobuf setter and an exception's own f-string suite: 725 passed, 5 skipped, 1 xfailed. ruff + mypy clean on src. * gw#640: post-mortem supervisor — report the death that happens below Python 0.56.1 instruments every death Python can see; run 9 produced six restarts and zero worker_fatal, so the remaining class is a signal (cgroup OOM SIGKILL, SIGSEGV, external kill). Fork a supervisor before the heavy imports: the parent stays tiny, outlives the worker, and reports WTERMSIG/WCOREDUMP plus memory.max/current/peak and the memory.events oom_kill delta through the existing worker_fatal carrier. A boot record covers whole-cgroup kills — the next boot reports the previous container death. Also: the host canary reported os.cpu_count() (the HOST's 32) next to a cgroup-derived ram_total_gb (the container's 14.9 GB). Report the cores this container may actually use, and benchmark with that many threads. * gw#640: prove the supervisor forwards SIGTERM (drain semantics unchanged) * pgw#648/#641-S2/#647-C1: per-device-group VRAM accounting, admission leases, single-flight per instance Three trains of the residency-v2 program (WORKER-RESIDENCY-DESIGN.md is the design of record). All real planner/admission/eviction logic under test; the only fake is torch.cuda (th#1105 doctrine). A. pgw#648 — DeviceGroup is the unit of placement. `_default_free_vram_bytes` SUMMED free VRAM across every CUDA device, so a 3x24GB pod reported 72GB free and admitted a 30GB model that fits on no single card. VRAM is not fungible between cards. `DeviceGroup(devices=...)` now owns the probe; one `Residency` accounts for exactly one group's pool and nothing here ever sums across groups. A group spanning devices (future TP mesh) sums its own members only — that is one placement unit by definition. B. pgw#641 Stage 2 — admission leases. `Residency.admit(sizes) -> Lease` replaces the gw#409 whole-job `executing()` pin on the run path. From admission on: (a) no eviction/demotion/release path may victim a leased ref — INCLUDING refs whose entries do not exist yet, which is the structural gap the pin no-op'd on (a freshly created entry was demotable between its track_vram and the execution-time pin); (b) bytes for not-yet-loaded refs are RESERVED, so two concurrent admissions can no longer book the same free VRAM and OOM each other mid-load. Two leases on one ref claim the MAX, not the sum — one future load, one claim — and a claim is consumed the moment track_vram books the real bytes. `make_room(for_refs=)` excludes the caller's own reservations (they are the demand being satisfied). `fits(sizes)` is the cheap honest "can this worker serve this now?" query the hub can eventually route on. Expected bytes follow the pgw#636 ask ladder: measured hint -> this dispatch's snapshot total -> banked snapshot -> 0. Admission never refuses here; the adaptive-fit ladder still owns overcommit. C. pgw#647 (C1) — single-flight per instance. One live instance == one binding set == one materialized graph with MUTABLE buffers (the resident LoRA branch; adapter enable state). Two concurrent requests on one instance corrupt each other. One-job-per-GPU masked this; `BoundedSemaphore(gpu_count)` on multi-GPU pods already does not. A per- _ClassRecord run gate now serializes adapter attach + handler + detach; jobs on DIFFERENT instances (different picks) still run concurrently. Acquired after the GPU permit and never held while acquiring one, so no cycle. `@endpoint(reentrant=True)` is the explicit opt-out for classes that mutate no instance state; it is a class-only declaration (a loose function holds no instance state to serialize) and rejects loudly on functions. tests/test_residency_v2.py is the program's one designed test home (pgw#645); later stages add rows there rather than new per-issue files. Verified in an isolated `git archive HEAD` overlay (the shared chaos worktree has 2+ other lanes mid-edit): 725 passed / 5 skipped / 1 xfailed, ruff + mypy clean. * th#1111: split runtime_ms into real stages — measured, classified, and reconciling runtime_ms was one opaque number over input fetch, text encode, denoise, VAE decode, image encode, credential stamp and upload; the GPU-permit wait was in no metric at all. Four fleet benchmarks ran without attributing a second. - stage_timing.py: per-request StageTimer. Exclusive nesting, per-thread stacks (uploads fan out), an interval log for window attribution. Renders JobMetrics.stage_ms with measured stages + total.prep / total.tail / total.denoise / class.{gpu_busy,small_gpu,gpu_idle,unattributed} / resid.{prep,tail,unattributed,overlap}. slot_prologue names the gap between runtime_ms's start and the handler window, so the map sums to runtime_ms EXACTLY rather than approximately. - denoise measured with no endpoint change: diffusers_step_callback marks every step end (un-throttled — the emit stays throttled), giving denoise total + denoise.step_mean on all 14 endpoints wired to the shared callback. - framework hooks: gpu_permit_wait and input_fetch (pre-handler, reported but outside the reconciliation), image_encode, video_encode, credential_stamp, upload (promotes _stream's finalize_elapsed_s). - ctx.stage("text_encode") for what only an endpoint knows. - class.gpu_busy / total.runtime is the per-request half of pgw#652's hot-fraction metric. - proto: JobMetrics.stage_ms = 15, additive; the vocabulary is documented at the field. Integration test drives the real worker over the hub-double with an image-shaped handler; the map reconciles with the wire's runtime_ms (675ms sample: text_encode 60 + denoise 161 + image_encode 386 + stamp 1 + resid.unattributed 67 = 675, exact). Un-bracketed decode time surfaces as resid.tail instead of vanishing. Unit tests cover the permit wait, exclusive nesting, the prologue and the concurrent-upload overlap case. * gw#640: dial the hub only for the deaths Python could not report (signal / whole-container), and isolate the boot record per boot-smoke run An ordinary non-zero exit already had 0.56.1's in-process worker_fatal, so the supervisor re-dialing it duplicated the row and added the ~7s report budget to every failing boot — which is what tipped the boot-smoke harness over its 25s budget. The shared default boot-record path also made one test report the previous test's death. * pgw#652: activation-aware admission — leases reserve the workspace a concurrent request allocates, learned from measured peaks Weights are not the whole cost of admitting a request. A 1024^2 diffusion request also holds GBs of latents and attention workspace, and the design's own warning is that "leases must reserve activation headroom PER CONCURRENT REQUEST or admission control will OOM the moment interleaving starts working". This builds that on the pgw#641 Stage 2 leases. Accounting. A `Lease` now carries an activation claim. Weight claims MAX per ref (two jobs needing one checkpoint share ONE future load); activation claims SUM across live leases (each concurrent request allocates its own workspace), and are never consumed — only released — because they are transient by definition. `fits()` counts the caller's own activation alongside its weights. `make_room()` subtracts OTHER in-flight requests' activation but not the calling lease's: that one is either already allocated (hence already visible in the measured free bytes) or is this job's own future demand, and double-counting it would evict a resident checkpoint on every load — exactly the pgw#636 pathology this program just removed. Learned, never declared (standing rule 2: no developer-facing residency knobs). `record_activation` takes `peak_vram_bytes` minus what was already allocated when the handler took the GPU — a measurement the executor already had in hand and threw away — and keeps a DECAYING high-water: up instantly in full (under-reserving is what OOMs) and bleeding 12.5% per subsequent request otherwise, so one 4096^2 outlier does not permanently tax residency depth for a stream of 512^2 requests. It is keyed by FUNCTION, not by pick: transient VRAM is a property of the shape and the graph, not of which checkpoint is bound, so a never-seen checkpoint inherits a real measurement instead of reserv…

…2.1) (#396) * gw#587/gw#605: pin — a fresh Executor (no setup yet) advertises pre-load candidate cell lookups Unit-level the candidates work (records are eagerly created at construction); the LIVE gap (4 consecutive boots never received the stored cell — every boot re-minted) is therefore in the candidate->StateDelta->release-config-> attach chain, not in cell_lookups() itself. Forensics + eliminated theories in tracker gw#605. * compile-cache: platform org _system -> root (lockstep with tensorhub th#940 chaos 0e4f6b1e) system_repo now derives root/family-<f>; ref parsing, self-mint publisher owner, docs, proto contract, and test fixtures move in lockstep. Only the org segment changes — repo/flavor grammar is byte-identical. * gw#608: disable the AOT autograd cache in the compile path — cell portability requires the FX cache as the lookup surface; pack asserts its FX entries before publish ROOT CAUSE (cracked dry, three same-key captures from three pods): the AOTAutogradCache key hashes fx_kwargs[get_decomp_fn] via the function REPR — which embeds the process memory address (ASLR), so AOT keys can NEVER match across processes. inductor/fxgraph entries were bit-identical across all pods (8/8); inductor/aotautograd keys had zero overlap. On the consumer, the AOT-layer miss recompiles without consulting the portable on-disk FX entries: cache_hits=0, cache_misses=8 on two independent hosts against a byte-good cell (fail-closed held — never eager). Fix: _disable_aot_autograd_cache() (env TORCHINDUCTOR_AUTOGRAD_CACHE=0 + torch._functorch.config.enable_autograd_cache=False) at capture_env/ seed_env AND apply() — symmetric for producer capture and consumer seeding. Lookups collapse to FxGraphCache (proven portable); cells shrink (~50% was AOT payload); fresh processes pay a cheap AOT re-analysis, not an inductor recompile. Hardening (coordinator ruling): a minting boot proves its EXECUTION — finish_fleet_mint now also proves its ARTIFACT: refuse to publish a capture with zero fxgraph entries, and (when the proof supplies its compiled-graph count via finalize_self_mint(expected_graphs=pipe_misses)) a partial capture. Revert-turns-red tests for both. (5 pre-existing chaos test failures reproduced on untouched HEAD.) * gw#608: adopt-test fake warmup writes the fxgraph store the artifact assertion requires (one entry per miss) — the 0.40.4 CI failure was this un-updated fake, not the capture path * th#960/pgw#609 Phase 2: tests/harness/ + P1-P7 (stream lifecycle, residency reconcile, slot binding precedence, CAS multiwriter, discovery lock, cancel/ backpressure, boot smoke) Harness extracted per design: hub_double.py generalizes FakeScheduler from test_worker_grpc_e2e.py (the only double, at the true hub gRPC boundary); blob_host.py is a real blake3-verified HTTP blob server; toy_endpoints.py consolidates toy fixtures (echo/stream/GPU-slot-yield/Hub-bound/Slot-bound/ billable); subprocess_runner.py generalizes the gw#591 boot-smoke pattern. P3 absorbs PR #353 (pgw#606/th#938) and PR #307 (pgw#583) as real hub-double dispatches (no mocking of gen_worker internals, unlike the original unit tests) - revert-verified red against lifecycle.py's pre-fix state, restored clean. P1/P2/P4/P5/P6/P7 absorb assertions from test_worker_grpc_e2e.py, test_shared_cas_root_multiwriter.py, and test_boot_smoke_gw591.py. Found + fixed in the harness itself: WorkerHarness must set the TENSORHUB_CACHE_DIR env var (gen_worker.models.cache_paths reads the process-wide get_settings() singleton, not the per-worker Settings instance) - passing tensorhub_cache_dir= to load_settings() alone silently no-ops, and every hub-double test defaults to a fresh per-call temp dir so downloaded bytes never leak between tests. 40 passed, 1 skipped (pgw#605 heartbeat: proto fields not generated yet) in ~33s local. Existing test files not touched or deleted (Phase 3). * th#960/pgw#609 Phase 2: P8 (convert publish contract) + P9 (result upload metrics), plus ruff cleanup + a hub_double(file_base_url=) knob P9 needs P8 (tests/convert/test_p8_convert_publish_contract.py, hermetic fake_hub): dtype-mismatch cast honesty (pgw#589/th#901, condensed from test_publish_as_is_dtype.py), classifier corpus table (pgw#593), and an xfail(strict=True) test-first pin for pgw#566 (open bug: normalize_adapter _state_dict only passes unet_config to converters with a NAMED parameter - diffusers' real SDXL lora_state_dict takes **kwargs only, so the SGM block remap silently never runs). pgw#569's W8A8 verifier ULP gate is a documented skip - no factored-out comparator to test hermetically without a real w8a8 artifact round trip. P9 (tests/test_p9_result_upload_metrics.py): inline vs blob_ref by size alone over a real hub-double + real local media-upload dedup sink (no S3 part scripting, matching test_media_upload_owner.py's pattern); JobMetrics typed usage survives both wire forms. Needed hub_double(file_base_url=) so the sink URL rides the FIRST HelloAck instead of a racy post-connect mutate. ruff clean (unused imports, one E731 lambda->named helper). mypy: src/gen_worker clean (untouched); new tests/harness + test files carry the same untyped- fixture-param convention as the rest of tests/ (mypy only gates src/gen_worker per CI, tests/ was never in scope). Full new suite (P1-P9 + P8's own file): 50 passed, 1 skipped (pgw#605, proto fields not generated), 1 xfailed (pgw#566, fix open) in ~31s local. * th#960/pgw#609 Phase 3: delete 4 test files fully superseded by P3/P4/P8 test_slot_boot_precedence_th938.py, test_model_slot_identity_gate.py -> superseded by test_p3_slot_binding_precedence.py (same assertions, over a real hub-double boundary instead of monkeypatched Executor.store internals). test_shared_cas_root_multiwriter.py -> superseded by test_p4_cas_multiwriter_integrity.py (same 3 tests reproduced verbatim + 1 new row). tests/convert/test_publish_as_is_dtype.py -> superseded by tests/convert/test_p8_convert_publish_contract.py (same 3 core tests against the same fake_hub). Deletions only, no other changes. Full deletion manifest (all 166 files audited, reasoning for what's NOT deleted) is in the th#960/pgw#609 tracker checkpoint - the bulk of the pre-existing suite tests production surface (CLI, LoRA runtime, quantization lanes, VRAM/RAM admission, media/upload, Slot/discovery SDK, tests/convert producer suite) that no P-test covers, so it stays pending explicit direction on the larger absorb-or-accept-loss effort that would be needed to delete it safely. * pgw#610: proto — measured per-tier DiskUsageReport on StateDelta (lockstep with th#962) * pgw#610: measured per-tier disk telemetry in every StateDelta (th#962 lockstep) - statvfs on the real mounts (CAS root = container tier; ismount-guarded fill source volume when attached; NFS tier reserved) — never a declared size - safely-reclaimable = ref-index bytes of DISK-tier refs inactive AND not in the desired set (the disk-GC LRU's eligible set); reuses ref-index bytes, no tree rescans - capacity_generation bumps only on a measured (64 MiB-quantized) shape change; rides Hello.state and every StateDelta; 30s periodic refresher is edge-suppressed * pgw#609/th#960 Phase 4: rewire CI to one suite, one lane Manual dispatch + PR-into-master only (drafts/doc-only paths run nothing); drop the on-push trigger. Same gates (mypy gating, ruff non-gating per debt note, HTTP-timeout guard, llama-server, uv build); pytest targets the whole tests/ dir so P-suite additions land automatically. * Revert "pgw#609/th#960 Phase 4: rewire CI to one suite, one lane" This reverts commit c39efb69b82d967393bd3f3243ce328d954380f7. * th#960/pgw#609 Phase 2b: absorb ~26 incident-pinned rows into P1/P2/P3/P5/ P6/P8/P9 + 2 new consolidated files, ahead of the bounded aggressive deletion sweep Bucket-level triage per the coordinator's ruling (Paul's directive: minimal greenfield set, ~92% removal). Absorbed only distinct bug-classes carrying a real pgw#/gw# incident number or a revert-catches-it story; untagged "tested thoroughly once" coverage goes without absorption (git history is the archive). P1 (+6): auth-rejection exit/retry-within-window, permanent-precondition fast-exit, hello-ack-deadline reconnect, not-leader redirect, hub-restart reconnect - absorbed from test_worker_grpc_e2e.py (#372) ahead of its deletion. P2 (+3): setup-failure FnUnavailable+recovery, host-RAM-failure wire ordering (th#807) from test_worker_grpc_e2e.py; corrupt-load quarantine+ retry-once (gw#408, J17 flood) from test_snapshot_corruption.py. P3 (+1): bare-ref prefetch binding resolution (#377) from test_executor_prefetch_binding.py. P5 (+1): heavy-dep stub contract (pgw#506) from test_discovery_heavy_deps.py. P6 (+1): SendQueue capacity-generation fencing from test_worker_grpc_e2e.py. P8 (+5): diffusers root-allinone skip (e2e J7 ENOSPC), standalone-component canonical-weight selection (gw#426), transformers sharded-index onnx/pickle exclusion, gguf explicit-quant refusal, non-safetensors-format refusal family - from test_classifier.py ahead of its deletion. P9 (+1): token-bound-owner routing (J19 run34) from test_media_upload_owner.py. New consolidated files (coordinator-authorized, one per lane with no P-test home): tests/test_quantization_lanes.py (W8A8/W4A4/FP8 contract + gw#409 promote-device-integrity, from test_w8a8.py/test_w4a4.py/ test_fp8_and_emergency_loading.py/test_promote_device_integrity.py) and tests/convert/test_convert_producer_contract.py (clone concurrency gw#442, disk preflight gw#462, download-skip bank th#592, publish resilience gw#462, from the tests/convert/ producer suite). Also reverted a premature local-only Phase 4 CI commit (c39efb6, from a prior session before an interruption) - never pushed, out of THIS issue's authorized scope ("do not touch CI yet"); preserved in history via revert, not discarded. Full new-suite gates: mypy clean, ruff clean, each new/changed file green in isolation. Deletion commit (the ~125 files these rows supersede) follows as a separate, deletions-only commit per the coordinator's mechanics. * th#960/pgw#609 Phase 3b: aggressive deletion sweep — 125 files (32,618 LoC) superseded or absorbed per the coordinator's ruling Every file here is either fully superseded by test_quantization_lanes.py / test_convert_producer_contract.py, absorbed as rows into P1/P2/P3/P5/P6/P8/ P9 (chaos b8274a1), or carries no incident pin (git history is the archive, per the coordinator's bound). Full manifest, bucket-by-bucket reasoning, and the dangling-import sweep are in the th#960/pgw#609 tracker Checkpoint 6, pushed before this commit. Deletions only, no other changes. Foreign-WIP clusters untouched: compile_cache/executor_adopt/fleet_cells (this issue's own claim-checkpoint note) and test_disk_telemetry_pgw610.py (landed on chaos by another lane during this session). Final state: 14 test files (3,293 LoC) + tests/harness/ (6 files) + conftest.py x2 + fake_hub.py, target was ~12-15 files per the coordinator. * pgw#610: lint — typed enum cast + drop unused import * pgw#610: CONTRACT.md — document StateDelta.disk_usage / DiskUsageReport * pgw#610: guard drain's disk-report cancel with getattr (stubbed Lifecycles skip __init__, same convention as _cancel_residency_reconcile) * gw#608 residual: AOT-cache disable must be process-global — torch 2.13 config user overrides are thread-local ContextVars Live disproof (0.40.5, B200, 2026-07-21): mint capture still packed 8 ASLR-keyed aotautograd entries and the store-served sibling still failed (warmups=1, cache_hits=0, cache_misses=8) — the 0.40.4 disable ran on the arming thread; the warmup compile thread saw the default True. The env var is read once at torch import (env_name_force), so post-import it is inert. Fix: entrypoint sets TORCHINDUCTOR_AUTOGRAD_CACHE=0 before any torch import (binds all threads + compile-worker subprocesses); _disable_aot_autograd_cache additionally mutates the installed config entry's env_value_force (process- global) for already-imported-torch embedders. Cross-thread revert-turns-red test added (fails on the 0.40.5 shape, passes with the fix). Dry evidence: fxgraph keys 8/8 identical across ALL five banked packs (0.40.1 x3 / 0.40.3 / 0.40.5); aotautograd level-2 keys 0/8 everywhere. samples/selfmint-proof/gw0405-run/. * th#964: family lane policy twin — bare conv-UNet (sd1/sd2/sdxl) bindings AUTO-pick scale-free #fp8 locally gen_worker.models.ladder gains family_root (modelfamily.Root twin) and the conv-UNet w8a8 exclusion table; a maybe_rebind_family_fp8 fold (hub-parity AUTO pick over the resolve's sibling_flavors, sm_89+ only, bf16 stays the sub-floor default; pins untouched) runs in the local CLI resolve ahead of the GGUF fold over one shared resolve. WorkerResolvedRepo carries the resolve's model_family (slot-declared family as fallback); select_gguf no longer lets an AUTO-ineligible w8a8 row suppress the GGUF pick. Shared vectors: none in this repo; no format change. * gw#613/th#965: universal app-level heartbeat — layer 2 of the 3-layer liveness contract (0.41.0) Hello declares heartbeat_interval_ms=30000; the pgw#610 disk-report task is promoted to the beat: it force-re-sends the full StateDelta every 30s from the asyncio event loop (the control loop that owes progress, never a detached thread), starts at startup() entry so boot hangs are covered, and keeps beating through drain until the stream closes. Edge-triggered deltas still flow as before; the beat is the same message re-sent unchanged. Contract §3 rewritten to the 3-layer model (lockstep tensorhub c03a531d). Tests: hello cadence, force-send bypasses edge suppression, beats flow while a startup coroutine is parked (the gw#612 shape), drain keeps beating until close. * gw#613 retune: 10s beat x 6 misses; disk stats measured every 30s riding every beat HEARTBEAT_INTERVAL_MS 30000 -> 10000 (hub reaps at 6 misses, ~60s). The pgw#610 statvfs/ref-index scan keeps its 30s measurement cadence behind a TTL cache so the 10s beat never turns it into a hot loop; the report still rides every StateDelta. Contract §3 event-loop discipline clause added — already true in 0.41.0: setup/warmup/compile/residency/GC all run off-loop (executor._to_thread_complete / asyncio.to_thread). * gw#612: gate self-mint publish on full capture coverage; finalize phase; run-26 root cause corrected (0.42.0) Run 26's filed post-seal_publish hang is disproven: the worker completed setup and advertised readiness; the wedge was the hub's singular compile fence treating the 2-lane record's same-identity self-attested targets as ambiguous (tensorhub lockstep fix). Real worker defect fixed: a shared mint capture missing an unexercised mandatory sibling's graphs was still published as the family cell, bricking every adopting boot (gw#611 qwen hits=1/misses=1). finalize_self_mint packs only; the executor publishes iff every capture sharer proved in, else withholds typed+loud. New 'finalize' activity phase covers the post-proof tail to readiness. * gw#613: soften run-26 narrative — app-silent, not proven-hung (gw#612 root cause: healthy-idle, hub fence starvation) * gw#613: marco_polo_wedge — event-loop-blocking liveness probe for th#965 layer-2 drills * gw#611: credit AOT-layer hits in the adopt warmup proof; calls= in the fail-closed detail; cross-process portability repro Measured mechanism for th#954's release-bricking 0/0: bundled AOT cache serves a fresh-process adopt with fxgraph counters fully silent, so a HEALTHY serving cell read cache_hits=0/cache_misses=0 and fail-closed. inductor_counters now surfaces aot_cache_hit/aot_cache_miss and the guard wrapper counts AOT hits as serving evidence (AOT stays pinned off per gw#608; a config regression now degrades to a proven boot, not a brick). Fail-closed detail carries calls= to split orphaned-wrapper (calls=0) from counter-blind serving (calls>0). New tests: real mint->pack->fresh- subprocess adopt asserts >=1 FX hit; bundled-AOT 0/0 mechanism pin; guard credits AOT hits (red on revert). Hub lockstep: tensorhub chaos 9e7dca8e quarantines cells failing their own adopt proof. * th#767: gen_worker.families.wan — WanDefaults (animegen lane) Second family after SdxlDefaults: steps/guidance/guidance_2/max_guidance for the Wan 2.2 A14B MoE envelope. Registered under tensorhub's canonical architecture root "wan22" (not a Compile(family=) shape-grid string) so repo-metadata PUT validates against the matching tensorhub schema (companion wan22.schema.json on tensorhub chaos). Lets the wan-2.2 endpoint's T2V function migrate to the Slot model and give AnimeGen-T2V its own distilled recipe (8 steps, guidance=1.0, guidance_2=1.0) via repo metadata instead of branching on checkpoint identity. * gw#614: synthesized media-modality warmup coverage (union family cells publish) + on_hello_ack model-set-diff cancel (A) _run_synthesized_warmup coverage pass: compile-target objects still unexercised after the planned jobs get media VARIANTS of the same base payloads (warmup.media_variants: one optional image/audio field filled with a generated asset, nothing else drifts) — input-routed sibling lanes (qwen edit) mint into the shared capture, so the gw#612 gate publishes the union cell instead of withholding forever; adopts prove the sibling lane instead of arming unproven. (B) th#961 defense in depth: a HelloAck whose semantic model set (resolutions+disk_refs+snapshots+hot) matches the running reconcile's target no longer cancels it (in-flight self_mint_compile survives benign plan rewrites); changed sets cancel as before. * gw#615: fix event-loop-blocking disk telemetry — the 0.40.7 LTX post-seal_publish hang pgw#610/th#962's disk_usage_report() ran statvfs()/stat() synchronously inline in Lifecycle._state_delta(), which many hot paths call directly with no event-loop offload — including right after a self-mint's seal_publish. The provider-attached VOLUME fill-source is a network-backed mount that can stall for minutes under load (exactly what a self-mint's weight download + cell pack produce). A stalled statvfs() there freezes the whole event loop: every StateDelta, RunJob dispatch, and even the th#965 heartbeat (same loop) stops until the syscall returns. ModelStore.disk_usage_report() now only reads a cache; the actual measurement (_measure_disk_usage_report) runs exclusively via refresh_disk_usage_report(), off-loop through asyncio.to_thread. Lifecycle kicks that refresh as a fire-and-forget background task (never awaited inline), gated to the existing ~30s TTL, so a stalled mount delays only its own telemetry freshness — never any send, dispatch, or heartbeat. Root-caused via git diff v0.40.5..v0.40.7 (executor.py/lifecycle.py) plus gw#612's hub-log forensics for the qwen case: gw#612 itself was hub-side (dispatch-fence starve, corrected root cause, no worker hang) and its worker-side companion fix addresses a different defect (poisoned family-cell publish on the NEXT boot). This is a distinct, genuinely-new gen-worker regression introduced by a70e7c9 (wave-2 telemetry), reproduced locally with no GPU via a stalled-statvfs monkeypatch. Tests: tests/test_disk_telemetry_pgw610.py (new test_disk_usage_report_never_measures_only_reads_the_cache, test_state_delta_never_blocks_event_loop_on_a_stalled_mount; existing tests updated to the cache/refresh split) + tests/test_heartbeat_gw613.py (new test_beats_flow_while_disk_measurement_is_stalled; existing TTL test updated to the fire-and-forget refresh). All revert-turns-red verified. Full suite 458 passed/5 skipped; ruff+mypy clean. * gw#608: FX-key forensics — a store-served proof failure names the diverging FxGraphHashDetails component; compile_seconds + per-object proof counts in the fail-closed detail * gw#608: clamp fx-divergence detail under the 2000-char activity error cap * gw#608: exhaustive fx_cache_failure_report — counts + same-key re-save sibling diff + load probes discriminate B1/B2 in one boot * gw#608 FIX: self-mint arm is transactional over the process-global cache env — no-target siblings decline before any dir mutation, arm failure restores the prior env, and a delivered-cell-seeded process never opens a capture (the store-served 8/8-miss root cause) * gw#585: consume tensorhub v4 private-input manifests — v3->v4 hard cut (PR #320 port onto chaos) Proto adopted byte-for-byte from tensorhub chaos 07e5520b (protocol 4, RunJob.input_assets=15 — 14 taken by th#913 lane — InputAsset fields 1-6; CLI NDJSON stays v1); pb2 regenerated with grpcio-tools 1.82.1. Worker resolves the exact ordered manifest via one bounded resolve call under the attempt-scoped capability, verifies size/BLAKE3/MIME/kind/decode, preserves opaque refs, sets only local_path, attempt-owned temp cleanup; endpoint build rejects Asset-bearing set/frozenset and non-string maps; deterministic traversal (declaration order, list index, sorted string map keys). Full local suite 558 passed; mypy/ruff/ timeout-lint clean. uv.lock: mechanical 0.42.0 catch-up to pyproject. * gw#585: marco-polo-attach — private-input echo probe (blake3/size/local_path, optional in-flight delay) for the v4 e2e lane * gw#585: GEN_WORKER_INTERNAL_OBJECT_HOSTS — exact-host allowlist for resolver-minted private-input URLs (datacenter-internal object stores); caller public transports keep the full SSRF gate * gw#585: marco-polo-attach reads via os.fspath (Asset is os.PathLike; no .path() method) * gw#608: fix publish-event race in the self-mint adopt test fake — publish() fills the published dict before signaling the event (CI KeyError: 'bytes') * pgw#617: hierarchical slot bindings — load base composition, substitute components from ModelBinding.components - proto: ModelBinding.components (field 5, map component -> canonical ref; mirrors tensorhub th#980); pb regenerated - ModelRef.component_overrides (tensorhub-only, sorted pairs) — part of the binding identity, so a component-only rebind derives a new instance/ residency identity; flat bindings byte-identical - executor: _effective_spec stamps overrides per dispatch; setup materializes each override ref from its own snapshot and injects the loaded module via the gw#479 components= from_pretrained mechanism; pins/held refs/digests/compile-cell binding facts include override refs - loading.load_component_override: class from the BASE model_index.json, weights from the override tree's subtree/root - typed ComponentSubstitutionError (unknown component, non-CAS ref, non-worker-loaded slot) - registry: selected_by= slots may omit default_checkpoint (th#980 relaxation mirror; unblocks ie#524 sweep of wan-2.2/sdxl/z-image) - tests: hub-double flat->substitute->flat identity proof, typed unknown- component refusal, registry relaxation (564 green) * pgw#617: mypy — dedicated slot_comps var in _effective_spec * gw#619: report a typed hardware-unsuitable fact to the hub before the boot-time CUDA probe's silent exit entrypoint.py's boot sequence (gw#529 cuda_probe) logged GEN_WORKER_CUDA_PROBE_FAILED and exit(1)'d with zero orchestrator contact — invisible to every layer of telemetry (th#986: ~20 pod deaths whose cause, CUDA-13 images on old-driver hosts per th#988, was unattributable). On a probe failure the worker now dials the hub and sends ONE HardwareUnsuitable WorkerMessage in place of Hello (cozy.scheduler wire addition, tensorhub th#988 companion commit) before exiting — bounded best-effort (~7s worst case: 2 attempts x 3s + 1s backoff), never a reason to delay the exit materially, and a silent fallback when the hub is unreachable or predates this field (old-hub FAILED_PRECONDITION rejection). cuda_probe.py: classify_probe_failure() — typed reason_class vocabulary (torch_unavailable | cuda_unavailable | driver_too_old | cuda_error | unknown) derived from CudaProbeResult.reason; driver_too_old matches torch's own "driver too old (found version ...)" message verbatim (the exact th#591/th#979/th#988 signature). hardware_report.py (new): builds the report (nvidia-smi/NVML for driver_version+gpu_name — reads the driver directly, independent of the broken CUDA runtime being reported on; torch_version/torch_cuda_version; gen_worker_version; image_digest/instance_id from Settings; worker_id/ release_id from Settings or JWT claims) and dials it over a one-shot grpc.aio Connect stream. entrypoint.py: calls report_hardware_unsuitable() right where the probe failure was previously silently fatal; logs a cuda_probe_hardware_report startup phase either way. Tests: tests/test_hardware_report.py (classify_probe_failure vocabulary; build_hardware_report degrades safely without nvidia-smi and reads driver identity when torch.cuda is down; report_hardware_unsuitable delivers to a real hub-double socket, resolves identity from JWT claims, and returns promptly-false against both an old-hub protocol rejection and a connection-refused unreachable hub). tests/test_boot_smoke_hardware_report.py (true end-to-end: a real `python -m gen_worker.entrypoint` subprocess whose probe fails on THIS box's actual driver/torch mismatch dials a real hub socket, and the hub-unreachable fallback still exits without hanging). tests/harness/hardware_report_hub.py: shared real-socket test servicers (recording_hub, old_hub, closed_port_addr). Full suite green (578 passed) plus the extended gw#591 boot-smoke contract (tests/test_boot_smoke_gw591.py, tests/test_p7_boot_smoke.py) confirmed unaffected. * ie#522: non-inference functions must never report loading_functions() _warmup_plan already excludes non-"inference" kinds from boot warmup (spec.kind != "inference"), but available_functions()/loading_functions() didn't share that gate: a declared-but-never-dispatched conversion function (a release can bundle many; only a few get invoked) sat in loading_functions() forever, since nothing was ever going to warm it. The hub's th#965 layer-3 stall watchdog takes loading_functions() at face value and kills a worker after 10 minutes of "no open activity" on any name in that list — including a name nothing was ever loading. Reproduced live: a wan-2.2 conversion release bundling z-image-w8a8-quantization alongside clone-huggingface got its pod killed at 10m15s three times running, on three different hosts, each time mid a legitimate multi-GB clone-huggingface transfer. Fix: conversion/training/dataset-kind functions are always available (matching the existing spec.slots per-dispatch-serveability carve-out just above) and never counted as loading. * ie#522: watchdog evidence blind to I/O-bound download+load phases The activity watchdog's default liveness evidence was process CPU seconds only. Two real activities are honestly alive while burning near-zero CPU: an I/O-bound network model fill (large composites, tens of GB, CPU-light by design), and the subsequent on-disk load into GPU (safetensors mmap + device placement — third-party diffusers/torch code with no progress hook). Both read as "stalled" to the hub's th#965 layer-3 rule after 10 minutes of no heartbeat, even mid genuine progress. Reproduced live twice: identical ~9.5min self_mint_compile CancelledError at phase=load, two different RunPod hosts. Fixed three ways: - _process_cpu_evidence now sums live (not just reaped) child-process CPU via psutil — covers inductor's async_compile subprocess workers, which resource.RUSAGE_CHILDREN can't see while they're still running. - note_progress() lets an I/O callback (model-download byte ticks) heartbeat the current activity directly, independent of the CPU-sampling watchdog thread; wired into cozy_snapshot.py's _on_bytes network-byte path. - _process_io_evidence adds process disk I/O bytes (psutil io_counters) to the default combined evidence — a universal, non-invasive signal for the on-disk load phase that needs no app-level hook into third-party loading code at all. Revert-turns-red verified: reverting activity.py/cozy_snapshot.py makes exactly the 3 new evidence tests fail (zero-CPU download, zero-CPU disk load, live-child-CPU), while the pre-existing dead- activity-still-stalls test keeps the detector honest. Full suite green (584 passed, 5 skipped, 1 xfailed). * gw#621: progress registry — named monotonic counters on the 10s beat + self-diagnosis New gen_worker.progress: phases register named monotonic counters (download:<ref> bytes w/ snapshot-known totals, evidence:<kind> riding the watchdog, warmup:jobs, infer:steps, upload:bytes); family prefix maps to a per-phase self-diagnosis window (code constants). activity.on_beat rides lifecycle's 10s heartbeat: one counter-carrying ActivityUpdate per beat — frozen counters included (the hub kills on counter non-advancement) — and self_stalled=true + stalled_for_ms when even the freshest counter is past its window (kill on confession, never CPU inference). Additive ActivityUpdate fields 10-16 (counter, unit, done, total, rate, self_stalled, stalled_for_ms; th#994 carries the proto source). Counters are activity- owned (finished at activity end). Downloads now carry bytes_total on the FIRST DOWNLOADING event when the snapshot knows sizes — no more 'started total=0' then silence. Also completes ie#522's 7640551, which swept the in-flight activity.py half of this work into its commit — gen_worker.progress ships here. * gw#621: tests — registry windows/self-diagnosis, counter beats, hub-double e2e confession Revert-turns-red for the ie#522 false-positive class: a CPU-quiet byte-advancing download is visibly alive on the wire (counter beats, never self_stalled) over a REAL Worker + hub double; a frozen counter still confesses, and local behavior is unchanged (the hub owns the kill). * gw#621: progress_beat re-checks _done — a beat racing completion never re-opens a terminal activity hub-side * ie#522: arm_compile() no-ops (never raises) with no active arming scope Endpoint setup() declares intent by calling arm_compile() unconditionally (sdxl, wan-2.2, ...); whether that intent is active is the release's decision — an eager registration (no compile=Compile(...) declared) already opens ArmingScope(None, ...) as a documented no-op, but arm_compile() itself still raised RuntimeError when called with no active scope. Reproduced live: wan-2.2's eager smoke (dropped compile= to skip a 51.5min self-mint compile grid per Paul's ruling) crashed at warmup/phase=load on this exact raise. Fix: no active scope -> log at info, return False (never armed), same as every other "stays eager" path. Revert-turns-red verified (4 tests updated/added in test_arm_compile_pgw517.py). Full suite green (594 passed, 5 skipped, 1 xfailed). * pgw(th#1004): @variant_of handler annotation — variant pairing emitted into the discovery manifest (variant_of/variant), dangling/self/chain targets fail the build * ie#522: WanDefaults gains an optional shift field (scheduler flow-match shift) Live finding #3: the AnimeGen card's own 8-step recipe pins FlowMatchEulerDiscreteScheduler(shift=3.0) alongside its distillation LoRA -- a step-distilled lineage needs a non-default scheduler shift, which nothing in the wan22 inference-defaults vocabulary carried. shift: Optional[float] = None -- every non-distilled repo's metadata and the resolved-defaults fallback leave the pipeline's own scheduler config untouched; only a repo that explicitly publishes shift gets one applied at the endpoint's resolution site (wan-2.2 main.py, separate commit). Mirrors tensorhub-chaos's wan22.schema.json shift addition. * ie#522 fix: de-flake the zero-cpu watchdog test for constrained CI runners test_watchdog_survives_zero_cpu_download_via_note_progress was flaky by construction: in its 0.3s real-clock window the 5s note_progress rate-limit floor allowed exactly ONE beat, so the >=2 assertion silently depended on incidental watchdog CPU/IO evidence noise for the second beat — present on a 32-core dev box, absent on 2-core GitHub runners (confirmed by per-thread beat attribution: ticker contributes 1, the rest come from the watchdog thread; beats=1 reproduces even locally outside pytest). Deterministic rewrite, intent intact (zero-CPU download must not go silent while note_progress ticks arrive): drop the rate-limit floor for this test (the floor is covered by its own test), pin watchdog evidence to a constant zero so every counted beat is attributable to note_progress alone (airtight revert-turns-red — verified: neutering note_progress fails it with 0 beats), and wait event-driven on beat count with a 10s deadline instead of a fixed wall-clock window. Verified 20x unconstrained, 10x taskset -c 0, 5x taskset -c 0,1; full file green; mypy clean. * pgw#622: eager-while-compiling with hot-swap — novel request shapes serve immediately A novel input signature at a compiled target no longer stalls 30-60s behind Dynamo+Inductor: the guard routes it to the eager original, one background thread warms the compiled callable with a zero-filled dummy batch (separate CUDA stream, nice +10), and a successful warm atomically hot-swaps the signature to the compiled path. Each completed warm repacks the live cache root and republishes the cell under the same key so the fleet never compiles that (shape, GPU, lane) again. Sequential compile-then-serve is kept for: the boot warmup-proof window, mandatory quantized lanes (w8a8/w4a4), tight VRAM headroom (degrade, never OOM), regional mode, and signature-vocabulary explosions. * release: gen-worker 0.45.0 — pgw#622 eager-while-compiling with hot-swap * release: 0.45.0 uv.lock version sync * th#1017: SdxlScheduler gains lcm/euler_trailing distilled-regime schedulers * th#1017: @endpoint regimes= decorator surface + manifest emission Per-method {name: (regime,...)} dict on class handlers (mirrors warmup=), bare tuple on the function form, default ("standard",). Validated against {standard,v_prediction,distilled}; EndpointSpec.regimes carries the resolved per-handler tuple; discover.py emits "regimes" only when it differs from the default (absent = ["standard"]). * th#1017: ResolvedSlot.regime + executor-level regime backstop resolve_slot/resolve_slots thread inference_regime (default "standard") into ResolvedSlot.regime, sourced from the hub resolve response's inference_regime field (hub_client.WorkerResolvedRepo, mirrors model_family; "" on hubs not sending it). RegimeMismatchError raises when a resolved checkpoint's regime is outside the invoked function's declared regimes= — wired as a backstop in executor._resolve_slots_kwargs via allowed_regimes=spec.regimes. * th#1017: converter stamps regime-correct scheduler config run_clone/build_flavor_tree accept an inference_regime hint (payload field of the same name); writer.apply_regime_scheduler_config rewrites a produced diffusers flavor's scheduler/config.json: v_prediction sets prediction_type + rescale_betas_zero_snr, distilled sets trailing timestep_spacing. No-op for standard or a tree with no scheduler component. Updated the P8 build_flavor_tree test stub to accept the new kwarg. * th#1017: tests — regimes decorator surface, manifest, resolution, converter Real discovery/registry/slot/convert codepaths (no mocking): decorator validation errors, class/function manifest round-trip + default-omitted, ResolvedSlot.regime + RegimeMismatchError backstop, hub_client inference_regime parsing, and scheduler-config stamping for all three regimes. * gw#623: only cancel an in-flight residency load when the ACTIVE model leaves the set Root cause of the ~9.5min (and ~15s-burst) warmup/self_mint_compile CancelledError at phase=load on nearly every serving boot: _semantic_model_key hashed full snapshot serializations INCLUDING presigned file URLs, which the hub re-signs on every ~15s release-config rebuild and deliberately excludes from its own HelloAck semantic hash. Every resent ack carrying rotated URLs (or any sibling-ref churn, e.g. the th#1021 preposition/seed fight) looked like a model-set change and killed the in-flight load. - _snapshot_content_key: digest + (path, size, blake3) per file; URL excluded - level-triggered reconcile: passes re-read the freshest desired set; a benign mid-load update signals a re-pass instead of cancelling - cancel only when the actively-loading item left the set or changed snapshot identity / precision resolution (_active_work_still_desired) Tests (revert-turns-red verified): URL rotation keeps the load, sibling ref added mid-load keeps the load then converges, active ref leaving cancels, active-ref resolution repick cancels. * gw#624: free cancelled load/warmup attempts' memory before the retry Live: 5 cancelled load retries on one worker climbed container RAM 3%->97% and VRAM to 83.86GB (OOM on an 80GB card) — each cancelled attempt's partially loaded modules stayed alive (pinned by the CancelledError's traceback and by uncollected reference cycles) while the retry loaded a fresh copy on top. - _to_thread_complete: drop the joined Task reference on the cancel path so the discarded load result is not pinned by the traceback frame during rollback - _injection_kwargs: release shared-component holds acquired before the failure (they never reached rec.shared_keys, so vacate could not) - _rollback_failed_setup: gc.collect + cuda.empty_cache after vacate, and schedule an allocation purge - _setup_locked: run the purge (gc + empty_cache) under the load lock BEFORE the next attempt allocates — a retry provably starts from baseline Tests (revert-turns-red verified): cancelled _to_thread_complete result is unreachable while the exception lives; cancelled ensure_setup's buffers are gone at the next attempt's setup entry (gc disabled for the duration). * release: gen-worker 0.46.0 — th#1017 inference regimes * th#1031: cell_selection_bug recovers via self-mint instead of retry-blocking forever A self-requested cell whose graph signature drifts from this runtime's own (cell_key has no graph-shape axis, so structurally different sdxl graphs can share one key) used to raise CellSelectionBugError straight out of fleet_cells.enable_compiled, aborting the whole arm attempt. For a mandatory w8a8/w4a4 lane that becomes CompiledLaneUnavailableError, which fails setup entirely and gets retried from scratch on the next request/job -- hitting the identical stale delivered cell again, forever. Live pod_events proves this: 54 cell_selection_bug rows across 2 releases/9 pods over 24h+ (predating pgw#622 by ~12h, so unrelated to it), all rejecting one stale signature (c3c4395265ad) that nothing was ever able to overwrite. ArmOutcome now carries the caught CellSelectionBugError; enable_compiled falls through to self-mint (the ordinary MISS recovery) instead of raising. The th#883 invariant stays wire-visible -- callers still send the cell_selection_bug ModelEvent/pod_event -- it just no longer blocks serving. Self-mint's own fail-closed safety net (no CUDA/toolchain/target) is untouched, so a genuine mint impossibility on a mandatory lane still refuses, now chained from the selection bug via __cause__. Threaded through both compile-arming seams: the worker-loaded slot path (_enable_compiled) and the self-loaded pipeline path (ArmingScope/ arm_compile, what sdxl actually uses) via a new selection_bugs map so neither silently drops the alarm. * th#1043: joint precision fit for shared-component multi-lane loads Sequential per-lane loading let the first lane in a gw#479 shared- component group (qwen-image's t2i/edit) consume free VRAM at native precision, starving the second lane into an offload placement the shared-component invariant refuses outright ("shared-component lanes require resident placement"). Decide fp8 storage for the whole group up front, against its combined footprint, before any lane loads. * th#1043: strict_vram refuses a load-time CUDA OOM instead of offloading place_pipeline's reactive OOM ladder had no strict_vram awareness: a CUDA OOM during placement would silently walk into model_offload even for a binding whose Resources(strict_vram=True) declares it cannot tolerate CPU-resident weights. Plan-time already refuses a PREDICTED offload under strict_vram, but a load-time OOM the plan didn't predict (e.g. a deploy-time binding that no longer matches the author's vram_gb sizing) reached the offload ladder anyway. Complements e4f9f6f's joint precision fit — defense in depth for any strict_vram binding, not just qwen-image's shared-component case. * th#1043: forced group-fit fp8 survives the bf16-resident upcast (0.48.1) (#375) Live 0.48.0 pod showed the joint fp8 forcing being silently un-done by load_from_pretrained's single-lane resident-upcast check (gw#534 rung 2): the first lane fit current free VRAM alone, upgraded back to bf16, and re-starved its sibling into the refused offload placement. A forced storage dtype now disables the local upgrade — the headroom belongs to the group, not the first lane to load. * th#1043: forced group-fit fp8 reports as a structural degradation (0.48.2) (#376) The joint shared-lane forcing served fp8 storage while the hub's plan said native bf16 — no FnDegraded, so placement never learns the card is too small for the group at native precision. Stamp the same rung outcome as the adaptive ladder. * th#1050: SDK lane contract — ctx.lane + opt-in handles= lane declaration (0.49.0) - ctx.lane: read-only executing lane on RequestContext, set post-degrade from the same truth as JobMetrics.lane; boot-warmup + CLI runs covered - @endpoint(handles=[...]): opt-in declared lane bodies (divergence marker, never inventory); validated against known concrete bodies; emitted into the discovery manifest as fn.handles - lane instruction: a declared lane is satisfiable with no laddered rebind (author code branches on ctx.lane); undeclared lanes refuse typed as before - hub-double e2e test: declared endpoint branches on ctx.lane over the real gRPC wire; JobMetrics.lane matches; undeclared dispatch unchanged * th#1051: RuntimeFormula SDK surface + worker-side term evaluation on the back-channel (0.50.0) * th#1051: uv.lock version bump * th#1051: lock gen-worker 0.50.0 * th#1051: mypy-clean Constant handling in formula eval/canonical * th#1055: desired-hot warm works on slot-only endpoints; refusals emit model events (0.50.1) ensure_desired_instance required the hot instance's binding set to equal spec.models, but deploy-bound Slots (ie#524/th#980) have no code default so spec.models is empty fleet-wide — every hub hot intent (gw#587 self-mint prewarm, th#912 seeding, #567 cell reload) failed a ValidationError that _reconcile_pass swallowed as one pod-local warning: no warmup, no self-mint, the th#868 w8a8 fence never opened, cells never armed. - validate against the declared slot set; code defaults may fill their own - declared-space hot bindings remap through the HelloAck precision picks - every desired-instance failure emits MODEL_STATE_FAILED for the instance refs (the silent stall is now impossible) * th#1055: version 0.50.2 (0.50.1 taken by pgw#626 on master) * th#1063: loud boot log when a datacenter pod has no CAS fill source (0.50.3) A RunPod pod without TENSORHUB_FILL_SOURCE_DIR (or with a non-mount path) pulls everything from R2 with write-through silently off — the defect that hid the fleet-wide unattached-volume regression. Boot now logs fill_source_disabled (reason=unset|not_a_mount) on datacenter pods and fill_source_enabled when the volume tier is live. * gw#627: Conv2d additive-branch support in the runtime LoRA overlay (0.51.0) Curated sdxl distill adapters (Lightning/DMD2) carry 49 conv LoRA pairs; the gw#547/558 branch was Linear-only, leaving raw peft as the only route on the w8a8 lane — fatal on _Fp8ScaledLinear (8/8 live turbo failures, th#1037 addendum). Plain nn.Conv2d denoiser modules are now branch-capable: canonical zeroed conv branches in the same rank bucket, staged-copy swap, eager instance-forward wrap conv1x1(conv(x,A),B). * pgw#628: residency reporting v2 — content-addressed idempotent observations (0.52.0) Worker half of th#1070: every applied HelloAck opens a republish epoch; the reconcile pass re-announces verified cached identities (exact ref+digest) once per epoch even when unchanged — a re-received plan (redrive/overdue resend/reconnect) is the hub asking for a resync, and success observations are safe to emit late, twice, or across plan revisions under the v2 hub's digest acceptance. Job-path cache hits within an epoch stay deduped. gw#614 same-set no-cancel unchanged. Revert-turns-red test; mypy+ruff clean; full suite 649 passed/5 skipped. No forced fleet rebuild — v2 hubs accept 0.44-0.51 reports fine. * gw#627 live fix: enable_compiled skips the branch lane on slot objects with no compile target A bare component slot (sdxl's AutoencoderKL vae) resolves none of cfg.targets; apply_lora_lane's no-denoiser error broke the whole model load (release-broken streak on the first gw627 deploy). Lane applies only to objects that resolve a compile target; the loud misconfig error stays for real targets. * gw#627 live fix 2: reject legacy attn-processor output from lora_state_dict normalization diffusers' non-diffusers converter turns kohya-flat sdxl attention keys into legacy …attn1.processor.to_q_lora.down.weight names that match no real module — the whole curated adapter failed typed (1680 unresolved keys, live request 88604466). Fall back to the raw keys, which resolve directly against module paths. gw627 test now runs the REAL SDXL converter in the loop. * th#1084 (pgw side): input-caused refusals raise ValidationError -> INVALID (0.53.0) RepoRefusal + SourceIncludeError are ValidationErrors; HF repo-access errors (RepositoryNotFound/Gated/Revision/EntryNotFound) wrap as typed input refusals at the convert ingest boundary with the class name preserved; an all-input-rejection 'no publishable flavor' aggregate is INVALID. Bad user repos fail only their request — never the release. * gw#631: delivered lora-bucket cell at a bare component slot is an ordinary miss, not a CellSelectionBugError r3 live chain (pod xzhlioakq909wq): the self-key check computed the family lora64 key with cfg.lora_bucket even when no cfg.target resolves on the slot object (sdxl's standalone vae), claimed the delivered cell as its own, and raised on lane drift (weight_lane 'lora64' != pipeline '') -> gw#608 seeded-cell self-mint refusal -> cell quarantine -> pod retired all_declared_functions_disabled. 0.52.1 downgraded apply_lora_lane for targetless slots; this applies the same effective-bucket rule to the self-key. * gw#632: renumber comments (id raced with slim-down filing) * th#1087 stages B+D: declared config params/envs (ConfigParam, env=), ctx.config, gen-push reconcile + local snapshot file for subprocesses (0.54.0) * th#1087: align worker config reconcile with final protocol * gw#631: remove unused worker and SDK surface * fix(config): pin subprocess snapshots per invocation * gw#631: retire unused H2D canary field * gw#496: remove silently dropped checkpoint metadata * gw#496: trim unused dataset conveniences * Project lifecycle shadow state from workers * Centralize worker lifecycle intent reporting * feat: report worker lifecycle intents * fix: restore RunJob org ownership contract * Implement exact config convergence protocol * marco-polo: regenerate uv.lock for editable gen-worker 0.56.0 * th#1098: marco-polo example 1.0.1 — gen-worker range to >=0.56,<0.57 (protocol v5), relock * pgw#636/#637/#638: hot-GPU mandate — pack VRAM with checkpoints, credit dynamo's in-memory cache, stop starving staged downloads pgw#637 (P0, the 2026-07-24 sdxl root cause). Cell keys are checkpoint-free, so the 2nd checkpoint of an already-proven family serves its warmup from dynamo's in-memory compiled code with ZERO FX/AOT counter movement (calls=1, hits=0, misses=0) and was deterministically judged "disproven" -> CompiledLaneUnavailableError -> FnUnavailable compile_cell_failed on every worker. The finalize proof now credits that signature, gated on BOTH: 1. compile_cache.cell_proven_in_process(ref) — this process actually served that exact cell (a real FX/AOT hit, or a finalized self-mint), and 2. compile_cache.has_inmemory_compiled_code(pipe) — DIRECT evidence from torch._dynamo that compiled code is live for THIS object's targets. Both are load-bearing: (1) alone lets a sibling object's hit certify this object's silence, which gw#603/gw#611 forbid (and which broke two guard tests); (2) alone would credit a cell never proven anywhere. Aggravator fixed: apply() tracks armed pipelines in a WeakSet and unwrap() skips the PROCESS-GLOBAL torch._dynamo.reset() while a sibling is still armed, so a failing 2nd arm no longer kills the healthy 1st checkpoint's lane. pgw#636 (Paul: "vram_gb means 'at least X', not 'only X'"). The one-pipeline ceiling was the pre-load ask: max(sum(vram_hints), Resources.vram_gb) reserved 12 GiB (+2 GiB margin) for every never-seen checkpoint pick and evicted the resident pipeline on every hop. _estimate_setup_need now prefers a prior MEASURED vram_hint, else the wire snapshot's real byte total, and falls back to the declaration only when a ref has no byte facts at all. Slot( share_components=...) opts a dynamic slot into cross-pick content-keyed component sharing (gw#479 generalized, content-honest: unequal bytes never alias). Residency shared entries move from refcount to `holders`: referenced components are demotable while idle (owners pin+promote before executing), never evictable while referenced, and 2+-holder entries sort LAST in the LRU; record vacate no longer eagerly drains them. pgw#638. Disk staging is network+disk I/O and holds no GPU lock, so _reconcile_pass no longer gates it on executor.wait_idle() and on_message(run_job) no longer cancels it (only an in-flight HOT warm is preempted). Live: staged 4.7 GB downloads sat at 0% for 4+ minutes on busy workers while the hub bought pods for models the fleet nearly had. The reconcile loop parks on wait_idle() OR restart after a pass on a busy worker, so a fresh HelloAck is never stuck behind a long tenant job. Tests (real code paths; fakes only at the torch/network/capacity boundary): - test_compile_proof_pgw637.py — proven-cell registry, the dynamo probe (negative and positive), unwrap sibling-scoping. - test_executor_adopt.py::test_second_checkpoint_served_from_dynamo_inmemory_cache_proves — the filed regression through the REAL ensure_setup finalize path, plus the same counter signature WITHOUT live dynamo code still failing closed. - test_hot_gpu_pgw636.py — real _estimate_setup_need, real Residency holders semantics and LRU ordering, the real make_room packing juggle (four ~5 GB picks hot on a 24 GiB budget, one LRU victim under real pressure, warm promote back), serve-while-downloading over the real worker + hub-double, and the end-to-end juggle: four checkpoints through ONE selected_by= slot, round two runs zero additional setups (all four instances retained). * pgw#639 (partial): SIGUSR2 -> faulthandler stack dump of every thread The worker's asyncio loop owns transport and the 10s heartbeat while all model work runs on threads, so a thread wedged inside a C extension keeps heartbeating and hub-side logs can never see it. Registering faulthandler on SIGUSR2 at boot makes 'kill -USR2 <pid>' print all thread stacks to stderr, allocation-free, so it also works under memory pressure. This is the filed 'trivial first step'. The delivery half (getting a signal into the pod: sshd behind an env flag, or a hub-mediated debug RPC) is still open on pgw#639. * th#1107: three serve-path fixes — vae_only stops taxing pipelines that fit, strict_vram binds the auto SELECTION, image path signals the finalize handoff vae_only is chosen when the pipeline FITS and only headroom is tight, yet it enabled VAE tiling + attention slicing (memory.py:672) — tiled decode re-runs the VAE per tile and attention slicing replaces the fused SDPA/flash kernel. Both are now reserved for the rungs where the model genuinely does not fit. strict_vram was enforced only inside the OOM-demotion branch, so select_auto_mode returning model_offload/group_offload placed CPU-resident weights with no OOM and no error — the th#1043 shape reaching serving as a silent 5-10x tax. write_image never called _release_gpu_slot_for_finalize, so gw#516's overlap was dead on every image endpoint: finalize_wall_ms == 0 on 19 of 20 endpoints in the master hub DB (979 sdxl samples), non-zero only on ltx-video (median 3.25s of a 14.9s request). The encode+upload tail now runs slotless like the video path. * gw#640/th#1077: worker fatals reach the hub, not just pod stdout entrypoint._log_worker_fatal wrote the exception class, message and traceback to the pod's stdout ONLY. RunPod exposes no container-logs API, so a cloud-only worker death was un-debuggable by construction — the th#1085 cold-boot investigation burned six live runs on a crash whose traceback existed and was unreachable. The fatal is now also dialed to the orchestrator over a fresh Connect stream, reusing the HardwareUnsuitable carrier with reason_class="worker_fatal". The hub already persists that message as a durable pod_events row (class + reason + full JSON payload), so this needs no proto change and no hub redeploy, and works against every hub pin already deployed. Second half: the run loop ending WITHOUT a hub Drain or a shutdown signal was a clean, silent `return 0` — exactly the gw#640 signature the hub could only observe as a young-worker death. That is now a reported UnexpectedWorkerExit fatal. Worker.run() keeps its int contract and reports from the sync wrapper; the cuda_probe path is left alone because it already sends its own typed HardwareUnsuitable report. * pgw#638: revert serve-while-downloading — the naive fix breaks the mutable-tag identity fence Dropping the tenant-idle gate on disk staging (and the run_job cancellation of the reconcile pass) is the obvious fix for the measured starvation, and it is wrong. The pass's first act for a ref is to fence a stale resident identity via revalidate_snapshot_identity; a tenant job may legitimately be re-materializing OLDER bytes for that same ref concurrently. Unserialized, the fence loses the race and the worker reports the stale identity indefinitely — the th#1066 hub/worker residency drift class. Evidence: test_p2_residency_reconcile.py:: test_mutable_tag_move_fences_events_by_digest_and_generation goes red and does not converge within 10s of polling. Two attempted patches also failed: - re-verify after every job — handle_run_job returns BEFORE the job's load completes, so the re-verify still races it; - gate only when a fence is actually pending (needs_identity_fence) — the predicate is evaluated before the job's record becomes ready, so it reads False and skips the gate exactly when it is needed. The correct fix is pgw#641 Stage 4: transfers become their own tasks with their own lifecycle, so staging never sits inside this serialized pass and never has to choose between progress and ordering. Recorded in pgw#638. pgw#636 and pgw#637 are unaffected and stay. Also drops the `hold` toy endpoint and the serve-while-downloading test that only existed for the reverted behavior; the four-checkpoint juggle test stays. * pgw#650: legacy sweep — delete the pre-slot ModelChoice binding vocabulary, the dead ar_tts runtime, and 12 zero-consumer symbols The model SET is catalog, not code (th#980/ie#524/th#1087): Model(id, Hub(...), defaults) rows embedded a curated checkpoint list in endpoint code, superseded by Slot(selected_by=) + hub mapping. Zero endpoints declare one; the model.choices[] manifest block has never been emitted in production. - delete api/model.py (Model/ModelChoice/ModelDefaults/is_model_choice) and discovery's _model_choice_in/_collect_model_placement_key + fn["model"] block - delete runtimes/ar_tts.py: zero importers, zero tests, zero docs, zero consumers; runtimes/__init__ advertised it in __all__ without importing it - delete 12 symbols with no reference in src, tests, or any consumer repo: resolve_slots, DestinationRepo, read_gguf_metadata, materialize_safetensors_input, is_calibration_required, is_inline_supported, spec_id, snapshot_weight_bytes, with_oom_retry, quantized_modules, content_credentials._reset_for_tests - tests: delete test_request_context_checkpoint_contract.py (inspect.signature only, executes no behavior), test_boot_smoke_gw591.py (assertion-identical duplicate of test_p7_boot_smoke.py, pays a second real torch subprocess boot per run), test_families_wan.py (WanDefaults has no consumer; resolve_slot precedence is covered by P3), test_testing_helpers.py (tests the test helper); drop two rows that assert a protobuf setter and an exception's own f-string suite: 725 passed, 5 skipped, 1 xfailed. ruff + mypy clean on src. * gw#640: post-mortem supervisor — report the death that happens below Python 0.56.1 instruments every death Python can see; run 9 produced six restarts and zero worker_fatal, so the remaining class is a signal (cgroup OOM SIGKILL, SIGSEGV, external kill). Fork a supervisor before the heavy imports: the parent stays tiny, outlives the worker, and reports WTERMSIG/WCOREDUMP plus memory.max/current/peak and the memory.events oom_kill delta through the existing worker_fatal carrier. A boot record covers whole-cgroup kills — the next boot reports the previous container death. Also: the host canary reported os.cpu_count() (the HOST's 32) next to a cgroup-derived ram_total_gb (the container's 14.9 GB). Report the cores this container may actually use, and benchmark with that many threads. * gw#640: prove the supervisor forwards SIGTERM (drain semantics unchanged) * pgw#648/#641-S2/#647-C1: per-device-group VRAM accounting, admission leases, single-flight per instance Three trains of the residency-v2 program (WORKER-RESIDENCY-DESIGN.md is the design of record). All real planner/admission/eviction logic under test; the only fake is torch.cuda (th#1105 doctrine). A. pgw#648 — DeviceGroup is the unit of placement. `_default_free_vram_bytes` SUMMED free VRAM across every CUDA device, so a 3x24GB pod reported 72GB free and admitted a 30GB model that fits on no single card. VRAM is not fungible between cards. `DeviceGroup(devices=...)` now owns the probe; one `Residency` accounts for exactly one group's pool and nothing here ever sums across groups. A group spanning devices (future TP mesh) sums its own members only — that is one placement unit by definition. B. pgw#641 Stage 2 — admission leases. `Residency.admit(sizes) -> Lease` replaces the gw#409 whole-job `executing()` pin on the run path. From admission on: (a) no eviction/demotion/release path may victim a leased ref — INCLUDING refs whose entries do not exist yet, which is the structural gap the pin no-op'd on (a freshly created entry was demotable between its track_vram and the execution-time pin); (b) bytes for not-yet-loaded refs are RESERVED, so two concurrent admissions can no longer book the same free VRAM and OOM each other mid-load. Two leases on one ref claim the MAX, not the sum — one future load, one claim — and a claim is consumed the moment track_vram books the real bytes. `make_room(for_refs=)` excludes the caller's own reservations (they are the demand being satisfied). `fits(sizes)` is the cheap honest "can this worker serve this now?" query the hub can eventually route on. Expected bytes follow the pgw#636 ask ladder: measured hint -> this dispatch's snapshot total -> banked snapshot -> 0. Admission never refuses here; the adaptive-fit ladder still owns overcommit. C. pgw#647 (C1) — single-flight per instance. One live instance == one binding set == one materialized graph with MUTABLE buffers (the resident LoRA branch; adapter enable state). Two concurrent requests on one instance corrupt each other. One-job-per-GPU masked this; `BoundedSemaphore(gpu_count)` on multi-GPU pods already does not. A per- _ClassRecord run gate now serializes adapter attach + handler + detach; jobs on DIFFERENT instances (different picks) still run concurrently. Acquired after the GPU permit and never held while acquiring one, so no cycle. `@endpoint(reentrant=True)` is the explicit opt-out for classes that mutate no instance state; it is a class-only declaration (a loose function holds no instance state to serialize) and rejects loudly on functions. tests/test_residency_v2.py is the program's one designed test home (pgw#645); later stages add rows there rather than new per-issue files. Verified in an isolated `git archive HEAD` overlay (the shared chaos worktree has 2+ other lanes mid-edit): 725 passed / 5 skipped / 1 xfailed, ruff + mypy clean. * th#1111: split runtime_ms into real stages — measured, classified, and reconciling runtime_ms was one opaque number over input fetch, text encode, denoise, VAE decode, image encode, credential stamp and upload; the GPU-permit wait was in no metric at all. Four fleet benchmarks ran without attributing a second. - stage_timing.py: per-request StageTimer. Exclusive nesting, per-thread stacks (uploads fan out), an interval log for window attribution. Renders JobMetrics.stage_ms with measured stages + total.prep / total.tail / total.denoise / class.{gpu_busy,small_gpu,gpu_idle,unattributed} / resid.{prep,tail,unattributed,overlap}. slot_prologue names the gap between runtime_ms's start and the handler window, so the map sums to runtime_ms EXACTLY rather than approximately. - denoise measured with no endpoint change: diffusers_step_callback marks every step end (un-throttled — the emit stays throttled), giving denoise total + denoise.step_mean on all 14 endpoints wired to the shared callback. - framework hooks: gpu_permit_wait and input_fetch (pre-handler, reported but outside the reconciliation), image_encode, video_encode, credential_stamp, upload (promotes _stream's finalize_elapsed_s). - ctx.stage("text_encode") for what only an endpoint knows. - class.gpu_busy / total.runtime is the per-request half of pgw#652's hot-fraction metric. - proto: JobMetrics.stage_ms = 15, additive; the vocabulary is documented at the field. Integration test drives the real worker over the hub-double with an image-shaped handler; the map reconciles with the wire's runtime_ms (675ms sample: text_encode 60 + denoise 161 + image_encode 386 + stamp 1 + resid.unattributed 67 = 675, exact). Un-bracketed decode time surfaces as resid.tail instead of vanishing. Unit tests cover the permit wait, exclusive nesting, the prologue and the concurrent-upload overlap case. * gw#640: dial the hub only for the deaths Python could not report (signal / whole-container), and isolate the boot record per boot-smoke run An ordinary non-zero exit already had 0.56.1's in-process worker_fatal, so the supervisor re-dialing it duplicated the row and added the ~7s report budget to every failing boot — which is what tipped the boot-smoke harness over its 25s budget. The shared default boot-record path also made one test report the previous test's death. * pgw#652: activation-aware admission — leases reserve the workspace a concurrent request allocates, learned from measured peaks Weights are not the whole cost of admitting a request. A 1024^2 diffusion request also holds GBs of latents and attention workspace, and the design's own warning is that "leases must reserve activation headroom PER CONCURRENT REQUEST or admission control will OOM the moment interleaving starts working". This builds that on the pgw#641 Stage 2 leases. Accounting. A `Lease` now carries an activation claim. Weight claims MAX per ref (two jobs needing one checkpoint share ONE future load); activation claims SUM across live leases (each concurrent request allocates its own workspace), and are never consumed — only released — because they are transient by definition. `fits()` counts the caller's own activation alongside its weights. `make_room()` subtracts OTHER in-flight requests' activation but not the calling lease's: that one is either already allocated (hence already visible in the measured free bytes) or is this job's own future demand, and double-counting it would evict a resident checkpoint on every load — exactly the pgw#636 pathology this program just removed. Learned, never declared (standing rule 2: no developer-facing residency knobs). `record_activation` takes `peak_vram_bytes` minus what was already allocated when the handler took the GPU — a measurement the executor already had in hand and threw away — and keeps a DECAYING high-water: up instantly in full (under-reserving is what OOMs) and bleeding 12.5% per subsequent request otherwise, so one 4096^2 outlier does not permanently tax residency depth for a stream of 512^2 requests. It is keyed by FUNCTION, not by pick: transient VRAM is a property of the shape and the graph, not of which checkpoint is bound, so a never-seen checkpoint inherits a real measureme…

release 0.47.0: th#1031 cell_selection_bug self-mint recovery

7c16ab6

PaulFidika marked this pull request as ready for review July 23, 2026 07:54

PaulFidika merged commit cacb302 into master Jul 23, 2026
1 check passed

PaulFidika deleted the release/th1031-v0.47.0 branch July 23, 2026 08:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

release 0.47.0: th#1031 cell_selection_bug self-mint recovery#372

release 0.47.0: th#1031 cell_selection_bug self-mint recovery#372
PaulFidika merged 1 commit into
masterfrom
release/th1031-v0.47.0

PaulFidika commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

PaulFidika commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant