release: gen-worker 0.50.0 — th#1050 lane contract + th#1051 RuntimeFormula#377
Merged
Conversation
…ion (0.49.0) - ctx.lane: read-only executing lane on RequestContext, set post-degrade from the same truth as JobMetrics.lane; boot-warmup + CLI runs covered - @endpoint(handles=[...]): opt-in declared lane bodies (divergence marker, never inventory); validated against known concrete bodies; emitted into the discovery manifest as fn.handles - lane instruction: a declared lane is satisfiable with no laddered rebind (author code branches on ctx.lane); undeclared lanes refuse typed as before - hub-double e2e test: declared endpoint branches on ctx.lane over the real gRPC wire; JobMetrics.lane matches; undeclared dispatch unchanged
…the back-channel (0.50.0)
PaulFidika
added a commit
that referenced
this pull request
Jul 24, 2026
…master) (#386) * gw#587/gw#605: pin — a fresh Executor (no setup yet) advertises pre-load candidate cell lookups Unit-level the candidates work (records are eagerly created at construction); the LIVE gap (4 consecutive boots never received the stored cell — every boot re-minted) is therefore in the candidate->StateDelta->release-config-> attach chain, not in cell_lookups() itself. Forensics + eliminated theories in tracker gw#605. * compile-cache: platform org _system -> root (lockstep with tensorhub th#940 chaos 0e4f6b1e) system_repo now derives root/family-<f>; ref parsing, self-mint publisher owner, docs, proto contract, and test fixtures move in lockstep. Only the org segment changes — repo/flavor grammar is byte-identical. * gw#608: disable the AOT autograd cache in the compile path — cell portability requires the FX cache as the lookup surface; pack asserts its FX entries before publish ROOT CAUSE (cracked dry, three same-key captures from three pods): the AOTAutogradCache key hashes fx_kwargs[get_decomp_fn] via the function REPR — which embeds the process memory address (ASLR), so AOT keys can NEVER match across processes. inductor/fxgraph entries were bit-identical across all pods (8/8); inductor/aotautograd keys had zero overlap. On the consumer, the AOT-layer miss recompiles without consulting the portable on-disk FX entries: cache_hits=0, cache_misses=8 on two independent hosts against a byte-good cell (fail-closed held — never eager). Fix: _disable_aot_autograd_cache() (env TORCHINDUCTOR_AUTOGRAD_CACHE=0 + torch._functorch.config.enable_autograd_cache=False) at capture_env/ seed_env AND apply() — symmetric for producer capture and consumer seeding. Lookups collapse to FxGraphCache (proven portable); cells shrink (~50% was AOT payload); fresh processes pay a cheap AOT re-analysis, not an inductor recompile. Hardening (coordinator ruling): a minting boot proves its EXECUTION — finish_fleet_mint now also proves its ARTIFACT: refuse to publish a capture with zero fxgraph entries, and (when the proof supplies its compiled-graph count via finalize_self_mint(expected_graphs=pipe_misses)) a partial capture. Revert-turns-red tests for both. (5 pre-existing chaos test failures reproduced on untouched HEAD.) * gw#608: adopt-test fake warmup writes the fxgraph store the artifact assertion requires (one entry per miss) — the 0.40.4 CI failure was this un-updated fake, not the capture path * th#960/pgw#609 Phase 2: tests/harness/ + P1-P7 (stream lifecycle, residency reconcile, slot binding precedence, CAS multiwriter, discovery lock, cancel/ backpressure, boot smoke) Harness extracted per design: hub_double.py generalizes FakeScheduler from test_worker_grpc_e2e.py (the only double, at the true hub gRPC boundary); blob_host.py is a real blake3-verified HTTP blob server; toy_endpoints.py consolidates toy fixtures (echo/stream/GPU-slot-yield/Hub-bound/Slot-bound/ billable); subprocess_runner.py generalizes the gw#591 boot-smoke pattern. P3 absorbs PR #353 (pgw#606/th#938) and PR #307 (pgw#583) as real hub-double dispatches (no mocking of gen_worker internals, unlike the original unit tests) - revert-verified red against lifecycle.py's pre-fix state, restored clean. P1/P2/P4/P5/P6/P7 absorb assertions from test_worker_grpc_e2e.py, test_shared_cas_root_multiwriter.py, and test_boot_smoke_gw591.py. Found + fixed in the harness itself: WorkerHarness must set the TENSORHUB_CACHE_DIR env var (gen_worker.models.cache_paths reads the process-wide get_settings() singleton, not the per-worker Settings instance) - passing tensorhub_cache_dir= to load_settings() alone silently no-ops, and every hub-double test defaults to a fresh per-call temp dir so downloaded bytes never leak between tests. 40 passed, 1 skipped (pgw#605 heartbeat: proto fields not generated yet) in ~33s local. Existing test files not touched or deleted (Phase 3). * th#960/pgw#609 Phase 2: P8 (convert publish contract) + P9 (result upload metrics), plus ruff cleanup + a hub_double(file_base_url=) knob P9 needs P8 (tests/convert/test_p8_convert_publish_contract.py, hermetic fake_hub): dtype-mismatch cast honesty (pgw#589/th#901, condensed from test_publish_as_is_dtype.py), classifier corpus table (pgw#593), and an xfail(strict=True) test-first pin for pgw#566 (open bug: normalize_adapter _state_dict only passes unet_config to converters with a NAMED parameter - diffusers' real SDXL lora_state_dict takes **kwargs only, so the SGM block remap silently never runs). pgw#569's W8A8 verifier ULP gate is a documented skip - no factored-out comparator to test hermetically without a real w8a8 artifact round trip. P9 (tests/test_p9_result_upload_metrics.py): inline vs blob_ref by size alone over a real hub-double + real local media-upload dedup sink (no S3 part scripting, matching test_media_upload_owner.py's pattern); JobMetrics typed usage survives both wire forms. Needed hub_double(file_base_url=) so the sink URL rides the FIRST HelloAck instead of a racy post-connect mutate. ruff clean (unused imports, one E731 lambda->named helper). mypy: src/gen_worker clean (untouched); new tests/harness + test files carry the same untyped- fixture-param convention as the rest of tests/ (mypy only gates src/gen_worker per CI, tests/ was never in scope). Full new suite (P1-P9 + P8's own file): 50 passed, 1 skipped (pgw#605, proto fields not generated), 1 xfailed (pgw#566, fix open) in ~31s local. * th#960/pgw#609 Phase 3: delete 4 test files fully superseded by P3/P4/P8 test_slot_boot_precedence_th938.py, test_model_slot_identity_gate.py -> superseded by test_p3_slot_binding_precedence.py (same assertions, over a real hub-double boundary instead of monkeypatched Executor.store internals). test_shared_cas_root_multiwriter.py -> superseded by test_p4_cas_multiwriter_integrity.py (same 3 tests reproduced verbatim + 1 new row). tests/convert/test_publish_as_is_dtype.py -> superseded by tests/convert/test_p8_convert_publish_contract.py (same 3 core tests against the same fake_hub). Deletions only, no other changes. Full deletion manifest (all 166 files audited, reasoning for what's NOT deleted) is in the th#960/pgw#609 tracker checkpoint - the bulk of the pre-existing suite tests production surface (CLI, LoRA runtime, quantization lanes, VRAM/RAM admission, media/upload, Slot/discovery SDK, tests/convert producer suite) that no P-test covers, so it stays pending explicit direction on the larger absorb-or-accept-loss effort that would be needed to delete it safely. * pgw#610: proto — measured per-tier DiskUsageReport on StateDelta (lockstep with th#962) * pgw#610: measured per-tier disk telemetry in every StateDelta (th#962 lockstep) - statvfs on the real mounts (CAS root = container tier; ismount-guarded fill source volume when attached; NFS tier reserved) — never a declared size - safely-reclaimable = ref-index bytes of DISK-tier refs inactive AND not in the desired set (the disk-GC LRU's eligible set); reuses ref-index bytes, no tree rescans - capacity_generation bumps only on a measured (64 MiB-quantized) shape change; rides Hello.state and every StateDelta; 30s periodic refresher is edge-suppressed * pgw#609/th#960 Phase 4: rewire CI to one suite, one lane Manual dispatch + PR-into-master only (drafts/doc-only paths run nothing); drop the on-push trigger. Same gates (mypy gating, ruff non-gating per debt note, HTTP-timeout guard, llama-server, uv build); pytest targets the whole tests/ dir so P-suite additions land automatically. * Revert "pgw#609/th#960 Phase 4: rewire CI to one suite, one lane" This reverts commit c39efb69b82d967393bd3f3243ce328d954380f7. * th#960/pgw#609 Phase 2b: absorb ~26 incident-pinned rows into P1/P2/P3/P5/ P6/P8/P9 + 2 new consolidated files, ahead of the bounded aggressive deletion sweep Bucket-level triage per the coordinator's ruling (Paul's directive: minimal greenfield set, ~92% removal). Absorbed only distinct bug-classes carrying a real pgw#/gw# incident number or a revert-catches-it story; untagged "tested thoroughly once" coverage goes without absorption (git history is the archive). P1 (+6): auth-rejection exit/retry-within-window, permanent-precondition fast-exit, hello-ack-deadline reconnect, not-leader redirect, hub-restart reconnect - absorbed from test_worker_grpc_e2e.py (#372) ahead of its deletion. P2 (+3): setup-failure FnUnavailable+recovery, host-RAM-failure wire ordering (th#807) from test_worker_grpc_e2e.py; corrupt-load quarantine+ retry-once (gw#408, J17 flood) from test_snapshot_corruption.py. P3 (+1): bare-ref prefetch binding resolution (#377) from test_executor_prefetch_binding.py. P5 (+1): heavy-dep stub contract (pgw#506) from test_discovery_heavy_deps.py. P6 (+1): SendQueue capacity-generation fencing from test_worker_grpc_e2e.py. P8 (+5): diffusers root-allinone skip (e2e J7 ENOSPC), standalone-component canonical-weight selection (gw#426), transformers sharded-index onnx/pickle exclusion, gguf explicit-quant refusal, non-safetensors-format refusal family - from test_classifier.py ahead of its deletion. P9 (+1): token-bound-owner routing (J19 run34) from test_media_upload_owner.py. New consolidated files (coordinator-authorized, one per lane with no P-test home): tests/test_quantization_lanes.py (W8A8/W4A4/FP8 contract + gw#409 promote-device-integrity, from test_w8a8.py/test_w4a4.py/ test_fp8_and_emergency_loading.py/test_promote_device_integrity.py) and tests/convert/test_convert_producer_contract.py (clone concurrency gw#442, disk preflight gw#462, download-skip bank th#592, publish resilience gw#462, from the tests/convert/ producer suite). Also reverted a premature local-only Phase 4 CI commit (c39efb6, from a prior session before an interruption) - never pushed, out of THIS issue's authorized scope ("do not touch CI yet"); preserved in history via revert, not discarded. Full new-suite gates: mypy clean, ruff clean, each new/changed file green in isolation. Deletion commit (the ~125 files these rows supersede) follows as a separate, deletions-only commit per the coordinator's mechanics. * th#960/pgw#609 Phase 3b: aggressive deletion sweep — 125 files (32,618 LoC) superseded or absorbed per the coordinator's ruling Every file here is either fully superseded by test_quantization_lanes.py / test_convert_producer_contract.py, absorbed as rows into P1/P2/P3/P5/P6/P8/ P9 (chaos b8274a1), or carries no incident pin (git history is the archive, per the coordinator's bound). Full manifest, bucket-by-bucket reasoning, and the dangling-import sweep are in the th#960/pgw#609 tracker Checkpoint 6, pushed before this commit. Deletions only, no other changes. Foreign-WIP clusters untouched: compile_cache/executor_adopt/fleet_cells (this issue's own claim-checkpoint note) and test_disk_telemetry_pgw610.py (landed on chaos by another lane during this session). Final state: 14 test files (3,293 LoC) + tests/harness/ (6 files) + conftest.py x2 + fake_hub.py, target was ~12-15 files per the coordinator. * pgw#610: lint — typed enum cast + drop unused import * pgw#610: CONTRACT.md — document StateDelta.disk_usage / DiskUsageReport * pgw#610: guard drain's disk-report cancel with getattr (stubbed Lifecycles skip __init__, same convention as _cancel_residency_reconcile) * gw#608 residual: AOT-cache disable must be process-global — torch 2.13 config user overrides are thread-local ContextVars Live disproof (0.40.5, B200, 2026-07-21): mint capture still packed 8 ASLR-keyed aotautograd entries and the store-served sibling still failed (warmups=1, cache_hits=0, cache_misses=8) — the 0.40.4 disable ran on the arming thread; the warmup compile thread saw the default True. The env var is read once at torch import (env_name_force), so post-import it is inert. Fix: entrypoint sets TORCHINDUCTOR_AUTOGRAD_CACHE=0 before any torch import (binds all threads + compile-worker subprocesses); _disable_aot_autograd_cache additionally mutates the installed config entry's env_value_force (process- global) for already-imported-torch embedders. Cross-thread revert-turns-red test added (fails on the 0.40.5 shape, passes with the fix). Dry evidence: fxgraph keys 8/8 identical across ALL five banked packs (0.40.1 x3 / 0.40.3 / 0.40.5); aotautograd level-2 keys 0/8 everywhere. samples/selfmint-proof/gw0405-run/. * th#964: family lane policy twin — bare conv-UNet (sd1/sd2/sdxl) bindings AUTO-pick scale-free #fp8 locally gen_worker.models.ladder gains family_root (modelfamily.Root twin) and the conv-UNet w8a8 exclusion table; a maybe_rebind_family_fp8 fold (hub-parity AUTO pick over the resolve's sibling_flavors, sm_89+ only, bf16 stays the sub-floor default; pins untouched) runs in the local CLI resolve ahead of the GGUF fold over one shared resolve. WorkerResolvedRepo carries the resolve's model_family (slot-declared family as fallback); select_gguf no longer lets an AUTO-ineligible w8a8 row suppress the GGUF pick. Shared vectors: none in this repo; no format change. * gw#613/th#965: universal app-level heartbeat — layer 2 of the 3-layer liveness contract (0.41.0) Hello declares heartbeat_interval_ms=30000; the pgw#610 disk-report task is promoted to the beat: it force-re-sends the full StateDelta every 30s from the asyncio event loop (the control loop that owes progress, never a detached thread), starts at startup() entry so boot hangs are covered, and keeps beating through drain until the stream closes. Edge-triggered deltas still flow as before; the beat is the same message re-sent unchanged. Contract §3 rewritten to the 3-layer model (lockstep tensorhub c03a531d). Tests: hello cadence, force-send bypasses edge suppression, beats flow while a startup coroutine is parked (the gw#612 shape), drain keeps beating until close. * gw#613 retune: 10s beat x 6 misses; disk stats measured every 30s riding every beat HEARTBEAT_INTERVAL_MS 30000 -> 10000 (hub reaps at 6 misses, ~60s). The pgw#610 statvfs/ref-index scan keeps its 30s measurement cadence behind a TTL cache so the 10s beat never turns it into a hot loop; the report still rides every StateDelta. Contract §3 event-loop discipline clause added — already true in 0.41.0: setup/warmup/compile/residency/GC all run off-loop (executor._to_thread_complete / asyncio.to_thread). * gw#612: gate self-mint publish on full capture coverage; finalize phase; run-26 root cause corrected (0.42.0) Run 26's filed post-seal_publish hang is disproven: the worker completed setup and advertised readiness; the wedge was the hub's singular compile fence treating the 2-lane record's same-identity self-attested targets as ambiguous (tensorhub lockstep fix). Real worker defect fixed: a shared mint capture missing an unexercised mandatory sibling's graphs was still published as the family cell, bricking every adopting boot (gw#611 qwen hits=1/misses=1). finalize_self_mint packs only; the executor publishes iff every capture sharer proved in, else withholds typed+loud. New 'finalize' activity phase covers the post-proof tail to readiness. * gw#613: soften run-26 narrative — app-silent, not proven-hung (gw#612 root cause: healthy-idle, hub fence starvation) * gw#613: marco_polo_wedge — event-loop-blocking liveness probe for th#965 layer-2 drills * gw#611: credit AOT-layer hits in the adopt warmup proof; calls= in the fail-closed detail; cross-process portability repro Measured mechanism for th#954's release-bricking 0/0: bundled AOT cache serves a fresh-process adopt with fxgraph counters fully silent, so a HEALTHY serving cell read cache_hits=0/cache_misses=0 and fail-closed. inductor_counters now surfaces aot_cache_hit/aot_cache_miss and the guard wrapper counts AOT hits as serving evidence (AOT stays pinned off per gw#608; a config regression now degrades to a proven boot, not a brick). Fail-closed detail carries calls= to split orphaned-wrapper (calls=0) from counter-blind serving (calls>0). New tests: real mint->pack->fresh- subprocess adopt asserts >=1 FX hit; bundled-AOT 0/0 mechanism pin; guard credits AOT hits (red on revert). Hub lockstep: tensorhub chaos 9e7dca8e quarantines cells failing their own adopt proof. * th#767: gen_worker.families.wan — WanDefaults (animegen lane) Second family after SdxlDefaults: steps/guidance/guidance_2/max_guidance for the Wan 2.2 A14B MoE envelope. Registered under tensorhub's canonical architecture root "wan22" (not a Compile(family=) shape-grid string) so repo-metadata PUT validates against the matching tensorhub schema (companion wan22.schema.json on tensorhub chaos). Lets the wan-2.2 endpoint's T2V function migrate to the Slot model and give AnimeGen-T2V its own distilled recipe (8 steps, guidance=1.0, guidance_2=1.0) via repo metadata instead of branching on checkpoint identity. * gw#614: synthesized media-modality warmup coverage (union family cells publish) + on_hello_ack model-set-diff cancel (A) _run_synthesized_warmup coverage pass: compile-target objects still unexercised after the planned jobs get media VARIANTS of the same base payloads (warmup.media_variants: one optional image/audio field filled with a generated asset, nothing else drifts) — input-routed sibling lanes (qwen edit) mint into the shared capture, so the gw#612 gate publishes the union cell instead of withholding forever; adopts prove the sibling lane instead of arming unproven. (B) th#961 defense in depth: a HelloAck whose semantic model set (resolutions+disk_refs+snapshots+hot) matches the running reconcile's target no longer cancels it (in-flight self_mint_compile survives benign plan rewrites); changed sets cancel as before. * gw#615: fix event-loop-blocking disk telemetry — the 0.40.7 LTX post-seal_publish hang pgw#610/th#962's disk_usage_report() ran statvfs()/stat() synchronously inline in Lifecycle._state_delta(), which many hot paths call directly with no event-loop offload — including right after a self-mint's seal_publish. The provider-attached VOLUME fill-source is a network-backed mount that can stall for minutes under load (exactly what a self-mint's weight download + cell pack produce). A stalled statvfs() there freezes the whole event loop: every StateDelta, RunJob dispatch, and even the th#965 heartbeat (same loop) stops until the syscall returns. ModelStore.disk_usage_report() now only reads a cache; the actual measurement (_measure_disk_usage_report) runs exclusively via refresh_disk_usage_report(), off-loop through asyncio.to_thread. Lifecycle kicks that refresh as a fire-and-forget background task (never awaited inline), gated to the existing ~30s TTL, so a stalled mount delays only its own telemetry freshness — never any send, dispatch, or heartbeat. Root-caused via git diff v0.40.5..v0.40.7 (executor.py/lifecycle.py) plus gw#612's hub-log forensics for the qwen case: gw#612 itself was hub-side (dispatch-fence starve, corrected root cause, no worker hang) and its worker-side companion fix addresses a different defect (poisoned family-cell publish on the NEXT boot). This is a distinct, genuinely-new gen-worker regression introduced by a70e7c9 (wave-2 telemetry), reproduced locally with no GPU via a stalled-statvfs monkeypatch. Tests: tests/test_disk_telemetry_pgw610.py (new test_disk_usage_report_never_measures_only_reads_the_cache, test_state_delta_never_blocks_event_loop_on_a_stalled_mount; existing tests updated to the cache/refresh split) + tests/test_heartbeat_gw613.py (new test_beats_flow_while_disk_measurement_is_stalled; existing TTL test updated to the fire-and-forget refresh). All revert-turns-red verified. Full suite 458 passed/5 skipped; ruff+mypy clean. * gw#608: FX-key forensics — a store-served proof failure names the diverging FxGraphHashDetails component; compile_seconds + per-object proof counts in the fail-closed detail * gw#608: clamp fx-divergence detail under the 2000-char activity error cap * gw#608: exhaustive fx_cache_failure_report — counts + same-key re-save sibling diff + load probes discriminate B1/B2 in one boot * gw#608 FIX: self-mint arm is transactional over the process-global cache env — no-target siblings decline before any dir mutation, arm failure restores the prior env, and a delivered-cell-seeded process never opens a capture (the store-served 8/8-miss root cause) * gw#585: consume tensorhub v4 private-input manifests — v3->v4 hard cut (PR #320 port onto chaos) Proto adopted byte-for-byte from tensorhub chaos 07e5520b (protocol 4, RunJob.input_assets=15 — 14 taken by th#913 lane — InputAsset fields 1-6; CLI NDJSON stays v1); pb2 regenerated with grpcio-tools 1.82.1. Worker resolves the exact ordered manifest via one bounded resolve call under the attempt-scoped capability, verifies size/BLAKE3/MIME/kind/decode, preserves opaque refs, sets only local_path, attempt-owned temp cleanup; endpoint build rejects Asset-bearing set/frozenset and non-string maps; deterministic traversal (declaration order, list index, sorted string map keys). Full local suite 558 passed; mypy/ruff/ timeout-lint clean. uv.lock: mechanical 0.42.0 catch-up to pyproject. * gw#585: marco-polo-attach — private-input echo probe (blake3/size/local_path, optional in-flight delay) for the v4 e2e lane * gw#585: GEN_WORKER_INTERNAL_OBJECT_HOSTS — exact-host allowlist for resolver-minted private-input URLs (datacenter-internal object stores); caller public transports keep the full SSRF gate * gw#585: marco-polo-attach reads via os.fspath (Asset is os.PathLike; no .path() method) * gw#608: fix publish-event race in the self-mint adopt test fake — publish() fills the published dict before signaling the event (CI KeyError: 'bytes') * pgw#617: hierarchical slot bindings — load base composition, substitute components from ModelBinding.components - proto: ModelBinding.components (field 5, map component -> canonical ref; mirrors tensorhub th#980); pb regenerated - ModelRef.component_overrides (tensorhub-only, sorted pairs) — part of the binding identity, so a component-only rebind derives a new instance/ residency identity; flat bindings byte-identical - executor: _effective_spec stamps overrides per dispatch; setup materializes each override ref from its own snapshot and injects the loaded module via the gw#479 components= from_pretrained mechanism; pins/held refs/digests/compile-cell binding facts include override refs - loading.load_component_override: class from the BASE model_index.json, weights from the override tree's subtree/root - typed ComponentSubstitutionError (unknown component, non-CAS ref, non-worker-loaded slot) - registry: selected_by= slots may omit default_checkpoint (th#980 relaxation mirror; unblocks ie#524 sweep of wan-2.2/sdxl/z-image) - tests: hub-double flat->substitute->flat identity proof, typed unknown- component refusal, registry relaxation (564 green) * pgw#617: mypy — dedicated slot_comps var in _effective_spec * gw#619: report a typed hardware-unsuitable fact to the hub before the boot-time CUDA probe's silent exit entrypoint.py's boot sequence (gw#529 cuda_probe) logged GEN_WORKER_CUDA_PROBE_FAILED and exit(1)'d with zero orchestrator contact — invisible to every layer of telemetry (th#986: ~20 pod deaths whose cause, CUDA-13 images on old-driver hosts per th#988, was unattributable). On a probe failure the worker now dials the hub and sends ONE HardwareUnsuitable WorkerMessage in place of Hello (cozy.scheduler wire addition, tensorhub th#988 companion commit) before exiting — bounded best-effort (~7s worst case: 2 attempts x 3s + 1s backoff), never a reason to delay the exit materially, and a silent fallback when the hub is unreachable or predates this field (old-hub FAILED_PRECONDITION rejection). cuda_probe.py: classify_probe_failure() — typed reason_class vocabulary (torch_unavailable | cuda_unavailable | driver_too_old | cuda_error | unknown) derived from CudaProbeResult.reason; driver_too_old matches torch's own "driver too old (found version ...)" message verbatim (the exact th#591/th#979/th#988 signature). hardware_report.py (new): builds the report (nvidia-smi/NVML for driver_version+gpu_name — reads the driver directly, independent of the broken CUDA runtime being reported on; torch_version/torch_cuda_version; gen_worker_version; image_digest/instance_id from Settings; worker_id/ release_id from Settings or JWT claims) and dials it over a one-shot grpc.aio Connect stream. entrypoint.py: calls report_hardware_unsuitable() right where the probe failure was previously silently fatal; logs a cuda_probe_hardware_report startup phase either way. Tests: tests/test_hardware_report.py (classify_probe_failure vocabulary; build_hardware_report degrades safely without nvidia-smi and reads driver identity when torch.cuda is down; report_hardware_unsuitable delivers to a real hub-double socket, resolves identity from JWT claims, and returns promptly-false against both an old-hub protocol rejection and a connection-refused unreachable hub). tests/test_boot_smoke_hardware_report.py (true end-to-end: a real `python -m gen_worker.entrypoint` subprocess whose probe fails on THIS box's actual driver/torch mismatch dials a real hub socket, and the hub-unreachable fallback still exits without hanging). tests/harness/hardware_report_hub.py: shared real-socket test servicers (recording_hub, old_hub, closed_port_addr). Full suite green (578 passed) plus the extended gw#591 boot-smoke contract (tests/test_boot_smoke_gw591.py, tests/test_p7_boot_smoke.py) confirmed unaffected. * ie#522: non-inference functions must never report loading_functions() _warmup_plan already excludes non-"inference" kinds from boot warmup (spec.kind != "inference"), but available_functions()/loading_functions() didn't share that gate: a declared-but-never-dispatched conversion function (a release can bundle many; only a few get invoked) sat in loading_functions() forever, since nothing was ever going to warm it. The hub's th#965 layer-3 stall watchdog takes loading_functions() at face value and kills a worker after 10 minutes of "no open activity" on any name in that list — including a name nothing was ever loading. Reproduced live: a wan-2.2 conversion release bundling z-image-w8a8-quantization alongside clone-huggingface got its pod killed at 10m15s three times running, on three different hosts, each time mid a legitimate multi-GB clone-huggingface transfer. Fix: conversion/training/dataset-kind functions are always available (matching the existing spec.slots per-dispatch-serveability carve-out just above) and never counted as loading. * ie#522: watchdog evidence blind to I/O-bound download+load phases The activity watchdog's default liveness evidence was process CPU seconds only. Two real activities are honestly alive while burning near-zero CPU: an I/O-bound network model fill (large composites, tens of GB, CPU-light by design), and the subsequent on-disk load into GPU (safetensors mmap + device placement — third-party diffusers/torch code with no progress hook). Both read as "stalled" to the hub's th#965 layer-3 rule after 10 minutes of no heartbeat, even mid genuine progress. Reproduced live twice: identical ~9.5min self_mint_compile CancelledError at phase=load, two different RunPod hosts. Fixed three ways: - _process_cpu_evidence now sums live (not just reaped) child-process CPU via psutil — covers inductor's async_compile subprocess workers, which resource.RUSAGE_CHILDREN can't see while they're still running. - note_progress() lets an I/O callback (model-download byte ticks) heartbeat the current activity directly, independent of the CPU-sampling watchdog thread; wired into cozy_snapshot.py's _on_bytes network-byte path. - _process_io_evidence adds process disk I/O bytes (psutil io_counters) to the default combined evidence — a universal, non-invasive signal for the on-disk load phase that needs no app-level hook into third-party loading code at all. Revert-turns-red verified: reverting activity.py/cozy_snapshot.py makes exactly the 3 new evidence tests fail (zero-CPU download, zero-CPU disk load, live-child-CPU), while the pre-existing dead- activity-still-stalls test keeps the detector honest. Full suite green (584 passed, 5 skipped, 1 xfailed). * gw#621: progress registry — named monotonic counters on the 10s beat + self-diagnosis New gen_worker.progress: phases register named monotonic counters (download:<ref> bytes w/ snapshot-known totals, evidence:<kind> riding the watchdog, warmup:jobs, infer:steps, upload:bytes); family prefix maps to a per-phase self-diagnosis window (code constants). activity.on_beat rides lifecycle's 10s heartbeat: one counter-carrying ActivityUpdate per beat — frozen counters included (the hub kills on counter non-advancement) — and self_stalled=true + stalled_for_ms when even the freshest counter is past its window (kill on confession, never CPU inference). Additive ActivityUpdate fields 10-16 (counter, unit, done, total, rate, self_stalled, stalled_for_ms; th#994 carries the proto source). Counters are activity- owned (finished at activity end). Downloads now carry bytes_total on the FIRST DOWNLOADING event when the snapshot knows sizes — no more 'started total=0' then silence. Also completes ie#522's 7640551, which swept the in-flight activity.py half of this work into its commit — gen_worker.progress ships here. * gw#621: tests — registry windows/self-diagnosis, counter beats, hub-double e2e confession Revert-turns-red for the ie#522 false-positive class: a CPU-quiet byte-advancing download is visibly alive on the wire (counter beats, never self_stalled) over a REAL Worker + hub double; a frozen counter still confesses, and local behavior is unchanged (the hub owns the kill). * gw#621: progress_beat re-checks _done — a beat racing completion never re-opens a terminal activity hub-side * ie#522: arm_compile() no-ops (never raises) with no active arming scope Endpoint setup() declares intent by calling arm_compile() unconditionally (sdxl, wan-2.2, ...); whether that intent is active is the release's decision — an eager registration (no compile=Compile(...) declared) already opens ArmingScope(None, ...) as a documented no-op, but arm_compile() itself still raised RuntimeError when called with no active scope. Reproduced live: wan-2.2's eager smoke (dropped compile= to skip a 51.5min self-mint compile grid per Paul's ruling) crashed at warmup/phase=load on this exact raise. Fix: no active scope -> log at info, return False (never armed), same as every other "stays eager" path. Revert-turns-red verified (4 tests updated/added in test_arm_compile_pgw517.py). Full suite green (594 passed, 5 skipped, 1 xfailed). * pgw(th#1004): @variant_of handler annotation — variant pairing emitted into the discovery manifest (variant_of/variant), dangling/self/chain targets fail the build * ie#522: WanDefaults gains an optional shift field (scheduler flow-match shift) Live finding #3: the AnimeGen card's own 8-step recipe pins FlowMatchEulerDiscreteScheduler(shift=3.0) alongside its distillation LoRA -- a step-distilled lineage needs a non-default scheduler shift, which nothing in the wan22 inference-defaults vocabulary carried. shift: Optional[float] = None -- every non-distilled repo's metadata and the resolved-defaults fallback leave the pipeline's own scheduler config untouched; only a repo that explicitly publishes shift gets one applied at the endpoint's resolution site (wan-2.2 main.py, separate commit). Mirrors tensorhub-chaos's wan22.schema.json shift addition. * ie#522 fix: de-flake the zero-cpu watchdog test for constrained CI runners test_watchdog_survives_zero_cpu_download_via_note_progress was flaky by construction: in its 0.3s real-clock window the 5s note_progress rate-limit floor allowed exactly ONE beat, so the >=2 assertion silently depended on incidental watchdog CPU/IO evidence noise for the second beat — present on a 32-core dev box, absent on 2-core GitHub runners (confirmed by per-thread beat attribution: ticker contributes 1, the rest come from the watchdog thread; beats=1 reproduces even locally outside pytest). Deterministic rewrite, intent intact (zero-CPU download must not go silent while note_progress ticks arrive): drop the rate-limit floor for this test (the floor is covered by its own test), pin watchdog evidence to a constant zero so every counted beat is attributable to note_progress alone (airtight revert-turns-red — verified: neutering note_progress fails it with 0 beats), and wait event-driven on beat count with a 10s deadline instead of a fixed wall-clock window. Verified 20x unconstrained, 10x taskset -c 0, 5x taskset -c 0,1; full file green; mypy clean. * pgw#622: eager-while-compiling with hot-swap — novel request shapes serve immediately A novel input signature at a compiled target no longer stalls 30-60s behind Dynamo+Inductor: the guard routes it to the eager original, one background thread warms the compiled callable with a zero-filled dummy batch (separate CUDA stream, nice +10), and a successful warm atomically hot-swaps the signature to the compiled path. Each completed warm repacks the live cache root and republishes the cell under the same key so the fleet never compiles that (shape, GPU, lane) again. Sequential compile-then-serve is kept for: the boot warmup-proof window, mandatory quantized lanes (w8a8/w4a4), tight VRAM headroom (degrade, never OOM), regional mode, and signature-vocabulary explosions. * release: gen-worker 0.45.0 — pgw#622 eager-while-compiling with hot-swap * release: 0.45.0 uv.lock version sync * th#1017: SdxlScheduler gains lcm/euler_trailing distilled-regime schedulers * th#1017: @endpoint regimes= decorator surface + manifest emission Per-method {name: (regime,...)} dict on class handlers (mirrors warmup=), bare tuple on the function form, default ("standard",). Validated against {standard,v_prediction,distilled}; EndpointSpec.regimes carries the resolved per-handler tuple; discover.py emits "regimes" only when it differs from the default (absent = ["standard"]). * th#1017: ResolvedSlot.regime + executor-level regime backstop resolve_slot/resolve_slots thread inference_regime (default "standard") into ResolvedSlot.regime, sourced from the hub resolve response's inference_regime field (hub_client.WorkerResolvedRepo, mirrors model_family; "" on hubs not sending it). RegimeMismatchError raises when a resolved checkpoint's regime is outside the invoked function's declared regimes= — wired as a backstop in executor._resolve_slots_kwargs via allowed_regimes=spec.regimes. * th#1017: converter stamps regime-correct scheduler config run_clone/build_flavor_tree accept an inference_regime hint (payload field of the same name); writer.apply_regime_scheduler_config rewrites a produced diffusers flavor's scheduler/config.json: v_prediction sets prediction_type + rescale_betas_zero_snr, distilled sets trailing timestep_spacing. No-op for standard or a tree with no scheduler component. Updated the P8 build_flavor_tree test stub to accept the new kwarg. * th#1017: tests — regimes decorator surface, manifest, resolution, converter Real discovery/registry/slot/convert codepaths (no mocking): decorator validation errors, class/function manifest round-trip + default-omitted, ResolvedSlot.regime + RegimeMismatchError backstop, hub_client inference_regime parsing, and scheduler-config stamping for all three regimes. * gw#623: only cancel an in-flight residency load when the ACTIVE model leaves the set Root cause of the ~9.5min (and ~15s-burst) warmup/self_mint_compile CancelledError at phase=load on nearly every serving boot: _semantic_model_key hashed full snapshot serializations INCLUDING presigned file URLs, which the hub re-signs on every ~15s release-config rebuild and deliberately excludes from its own HelloAck semantic hash. Every resent ack carrying rotated URLs (or any sibling-ref churn, e.g. the th#1021 preposition/seed fight) looked like a model-set change and killed the in-flight load. - _snapshot_content_key: digest + (path, size, blake3) per file; URL excluded - level-triggered reconcile: passes re-read the freshest desired set; a benign mid-load update signals a re-pass instead of cancelling - cancel only when the actively-loading item left the set or changed snapshot identity / precision resolution (_active_work_still_desired) Tests (revert-turns-red verified): URL rotation keeps the load, sibling ref added mid-load keeps the load then converges, active ref leaving cancels, active-ref resolution repick cancels. * gw#624: free cancelled load/warmup attempts' memory before the retry Live: 5 cancelled load retries on one worker climbed container RAM 3%->97% and VRAM to 83.86GB (OOM on an 80GB card) — each cancelled attempt's partially loaded modules stayed alive (pinned by the CancelledError's traceback and by uncollected reference cycles) while the retry loaded a fresh copy on top. - _to_thread_complete: drop the joined Task reference on the cancel path so the discarded load result is not pinned by the traceback frame during rollback - _injection_kwargs: release shared-component holds acquired before the failure (they never reached rec.shared_keys, so vacate could not) - _rollback_failed_setup: gc.collect + cuda.empty_cache after vacate, and schedule an allocation purge - _setup_locked: run the purge (gc + empty_cache) under the load lock BEFORE the next attempt allocates — a retry provably starts from baseline Tests (revert-turns-red verified): cancelled _to_thread_complete result is unreachable while the exception lives; cancelled ensure_setup's buffers are gone at the next attempt's setup entry (gc disabled for the duration). * release: gen-worker 0.46.0 — th#1017 inference regimes * th#1031: cell_selection_bug recovers via self-mint instead of retry-blocking forever A self-requested cell whose graph signature drifts from this runtime's own (cell_key has no graph-shape axis, so structurally different sdxl graphs can share one key) used to raise CellSelectionBugError straight out of fleet_cells.enable_compiled, aborting the whole arm attempt. For a mandatory w8a8/w4a4 lane that becomes CompiledLaneUnavailableError, which fails setup entirely and gets retried from scratch on the next request/job -- hitting the identical stale delivered cell again, forever. Live pod_events proves this: 54 cell_selection_bug rows across 2 releases/9 pods over 24h+ (predating pgw#622 by ~12h, so unrelated to it), all rejecting one stale signature (c3c4395265ad) that nothing was ever able to overwrite. ArmOutcome now carries the caught CellSelectionBugError; enable_compiled falls through to self-mint (the ordinary MISS recovery) instead of raising. The th#883 invariant stays wire-visible -- callers still send the cell_selection_bug ModelEvent/pod_event -- it just no longer blocks serving. Self-mint's own fail-closed safety net (no CUDA/toolchain/target) is untouched, so a genuine mint impossibility on a mandatory lane still refuses, now chained from the selection bug via __cause__. Threaded through both compile-arming seams: the worker-loaded slot path (_enable_compiled) and the self-loaded pipeline path (ArmingScope/ arm_compile, what sdxl actually uses) via a new selection_bugs map so neither silently drops the alarm. * th#1043: joint precision fit for shared-component multi-lane loads Sequential per-lane loading let the first lane in a gw#479 shared- component group (qwen-image's t2i/edit) consume free VRAM at native precision, starving the second lane into an offload placement the shared-component invariant refuses outright ("shared-component lanes require resident placement"). Decide fp8 storage for the whole group up front, against its combined footprint, before any lane loads. * th#1043: strict_vram refuses a load-time CUDA OOM instead of offloading place_pipeline's reactive OOM ladder had no strict_vram awareness: a CUDA OOM during placement would silently walk into model_offload even for a binding whose Resources(strict_vram=True) declares it cannot tolerate CPU-resident weights. Plan-time already refuses a PREDICTED offload under strict_vram, but a load-time OOM the plan didn't predict (e.g. a deploy-time binding that no longer matches the author's vram_gb sizing) reached the offload ladder anyway. Complements e4f9f6f's joint precision fit — defense in depth for any strict_vram binding, not just qwen-image's shared-component case. * th#1043: forced group-fit fp8 survives the bf16-resident upcast (0.48.1) (#375) Live 0.48.0 pod showed the joint fp8 forcing being silently un-done by load_from_pretrained's single-lane resident-upcast check (gw#534 rung 2): the first lane fit current free VRAM alone, upgraded back to bf16, and re-starved its sibling into the refused offload placement. A forced storage dtype now disables the local upgrade — the headroom belongs to the group, not the first lane to load. * th#1043: forced group-fit fp8 reports as a structural degradation (0.48.2) (#376) The joint shared-lane forcing served fp8 storage while the hub's plan said native bf16 — no FnDegraded, so placement never learns the card is too small for the group at native precision. Stamp the same rung outcome as the adaptive ladder. * th#1050: SDK lane contract — ctx.lane + opt-in handles= lane declaration (0.49.0) - ctx.lane: read-only executing lane on RequestContext, set post-degrade from the same truth as JobMetrics.lane; boot-warmup + CLI runs covered - @endpoint(handles=[...]): opt-in declared lane bodies (divergence marker, never inventory); validated against known concrete bodies; emitted into the discovery manifest as fn.handles - lane instruction: a declared lane is satisfiable with no laddered rebind (author code branches on ctx.lane); undeclared lanes refuse typed as before - hub-double e2e test: declared endpoint branches on ctx.lane over the real gRPC wire; JobMetrics.lane matches; undeclared dispatch unchanged * th#1051: RuntimeFormula SDK surface + worker-side term evaluation on the back-channel (0.50.0) * th#1051: uv.lock version bump * th#1051: lock gen-worker 0.50.0 * th#1051: mypy-clean Constant handling in formula eval/canonical * th#1055: desired-hot warm works on slot-only endpoints; refusals emit model events (0.50.1) ensure_desired_instance required the hot instance's binding set to equal spec.models, but deploy-bound Slots (ie#524/th#980) have no code default so spec.models is empty fleet-wide — every hub hot intent (gw#587 self-mint prewarm, th#912 seeding, #567 cell reload) failed a ValidationError that _reconcile_pass swallowed as one pod-local warning: no warmup, no self-mint, the th#868 w8a8 fence never opened, cells never armed. - validate against the declared slot set; code defaults may fill their own - declared-space hot bindings remap through the HelloAck precision picks - every desired-instance failure emits MODEL_STATE_FAILED for the instance refs (the silent stall is now impossible) * th#1055: version 0.50.2 (0.50.1 taken by pgw#626 on master) * th#1063: loud boot log when a datacenter pod has no CAS fill source (0.50.3) A RunPod pod without TENSORHUB_FILL_SOURCE_DIR (or with a non-mount path) pulls everything from R2 with write-through silently off — the defect that hid the fleet-wide unattached-volume regression. Boot now logs fill_source_disabled (reason=unset|not_a_mount) on datacenter pods and fill_source_enabled when the volume tier is live. * gw#627: Conv2d additive-branch support in the runtime LoRA overlay (0.51.0) Curated sdxl distill adapters (Lightning/DMD2) carry 49 conv LoRA pairs; the gw#547/558 branch was Linear-only, leaving raw peft as the only route on the w8a8 lane — fatal on _Fp8ScaledLinear (8/8 live turbo failures, th#1037 addendum). Plain nn.Conv2d denoiser modules are now branch-capable: canonical zeroed conv branches in the same rank bucket, staged-copy swap, eager instance-forward wrap conv1x1(conv(x,A),B). * pgw#628: residency reporting v2 — content-addressed idempotent observations (0.52.0) Worker half of th#1070: every applied HelloAck opens a republish epoch; the reconcile pass re-announces verified cached identities (exact ref+digest) once per epoch even when unchanged — a re-received plan (redrive/overdue resend/reconnect) is the hub asking for a resync, and success observations are safe to emit late, twice, or across plan revisions under the v2 hub's digest acceptance. Job-path cache hits within an epoch stay deduped. gw#614 same-set no-cancel unchanged. Revert-turns-red test; mypy+ruff clean; full suite 649 passed/5 skipped. No forced fleet rebuild — v2 hubs accept 0.44-0.51 reports fine. * gw#627 live fix: enable_compiled skips the branch lane on slot objects with no compile target A bare component slot (sdxl's AutoencoderKL vae) resolves none of cfg.targets; apply_lora_lane's no-denoiser error broke the whole model load (release-broken streak on the first gw627 deploy). Lane applies only to objects that resolve a compile target; the loud misconfig error stays for real targets. * gw#627 live fix 2: reject legacy attn-processor output from lora_state_dict normalization diffusers' non-diffusers converter turns kohya-flat sdxl attention keys into legacy …attn1.processor.to_q_lora.down.weight names that match no real module — the whole curated adapter failed typed (1680 unresolved keys, live request 88604466). Fall back to the raw keys, which resolve directly against module paths. gw627 test now runs the REAL SDXL converter in the loop. * th#1084 (pgw side): input-caused refusals raise ValidationError -> INVALID (0.53.0) RepoRefusal + SourceIncludeError are ValidationErrors; HF repo-access errors (RepositoryNotFound/Gated/Revision/EntryNotFound) wrap as typed input refusals at the convert ingest boundary with the class name preserved; an all-input-rejection 'no publishable flavor' aggregate is INVALID. Bad user repos fail only their request — never the release. * gw#631: delivered lora-bucket cell at a bare component slot is an ordinary miss, not a CellSelectionBugError r3 live chain (pod xzhlioakq909wq): the self-key check computed the family lora64 key with cfg.lora_bucket even when no cfg.target resolves on the slot object (sdxl's standalone vae), claimed the delivered cell as its own, and raised on lane drift (weight_lane 'lora64' != pipeline '') -> gw#608 seeded-cell self-mint refusal -> cell quarantine -> pod retired all_declared_functions_disabled. 0.52.1 downgraded apply_lora_lane for targetless slots; this applies the same effective-bucket rule to the self-key. * gw#632: renumber comments (id raced with slim-down filing) * th#1087 stages B+D: declared config params/envs (ConfigParam, env=), ctx.config, gen-push reconcile + local snapshot file for subprocesses (0.54.0) * th#1087: align worker config reconcile with final protocol * gw#631: remove unused worker and SDK surface * fix(config): pin subprocess snapshots per invocation * gw#631: retire unused H2D canary field * gw#496: remove silently dropped checkpoint metadata * gw#496: trim unused dataset conveniences * Project lifecycle shadow state from workers * Centralize worker lifecycle intent reporting * feat: report worker lifecycle intents * fix: restore RunJob org ownership contract * Implement exact config convergence protocol * marco-polo: regenerate uv.lock for editable gen-worker 0.56.0 * fix CI gates: type _hf_access_error_classes for except-clause mypy; drop unused Sequence import
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotion train (two lanes complete-on-chaos):
Hub-side consumers land in the tensorhub th#1051 train (manifest parse, per-cell RLS constants, frontier quoting). SDK additions are inert against older hubs (unknown manifest keys ignored; runtime_terms map optional).
Local: formula/decorator/registry/discovery suites green in the chaos venv; full-suite local run blocked by a box-local torch cu130 wheel hash mismatch (download corruption on this host, not the lock) — CI is the gate.