release: gen-worker 0.42.0 (gw#615 serving-safety fix + gw#611/612/614 + wan family)#363
Merged
Conversation
…oad candidate cell lookups Unit-level the candidates work (records are eagerly created at construction); the LIVE gap (4 consecutive boots never received the stored cell — every boot re-minted) is therefore in the candidate->StateDelta->release-config-> attach chain, not in cell_lookups() itself. Forensics + eliminated theories in tracker gw#605.
…th#940 chaos 0e4f6b1e) system_repo now derives root/family-<f>; ref parsing, self-mint publisher owner, docs, proto contract, and test fixtures move in lockstep. Only the org segment changes — repo/flavor grammar is byte-identical.
…tability requires the FX cache as the lookup surface; pack asserts its FX entries before publish ROOT CAUSE (cracked dry, three same-key captures from three pods): the AOTAutogradCache key hashes fx_kwargs[get_decomp_fn] via the function REPR — which embeds the process memory address (ASLR), so AOT keys can NEVER match across processes. inductor/fxgraph entries were bit-identical across all pods (8/8); inductor/aotautograd keys had zero overlap. On the consumer, the AOT-layer miss recompiles without consulting the portable on-disk FX entries: cache_hits=0, cache_misses=8 on two independent hosts against a byte-good cell (fail-closed held — never eager). Fix: _disable_aot_autograd_cache() (env TORCHINDUCTOR_AUTOGRAD_CACHE=0 + torch._functorch.config.enable_autograd_cache=False) at capture_env/ seed_env AND apply() — symmetric for producer capture and consumer seeding. Lookups collapse to FxGraphCache (proven portable); cells shrink (~50% was AOT payload); fresh processes pay a cheap AOT re-analysis, not an inductor recompile. Hardening (coordinator ruling): a minting boot proves its EXECUTION — finish_fleet_mint now also proves its ARTIFACT: refuse to publish a capture with zero fxgraph entries, and (when the proof supplies its compiled-graph count via finalize_self_mint(expected_graphs=pipe_misses)) a partial capture. Revert-turns-red tests for both. (5 pre-existing chaos test failures reproduced on untouched HEAD.)
# Conflicts: # tests/test_fleet_cells.py
…assertion requires (one entry per miss) — the 0.40.4 CI failure was this un-updated fake, not the capture path
…idency reconcile, slot binding precedence, CAS multiwriter, discovery lock, cancel/ backpressure, boot smoke) Harness extracted per design: hub_double.py generalizes FakeScheduler from test_worker_grpc_e2e.py (the only double, at the true hub gRPC boundary); blob_host.py is a real blake3-verified HTTP blob server; toy_endpoints.py consolidates toy fixtures (echo/stream/GPU-slot-yield/Hub-bound/Slot-bound/ billable); subprocess_runner.py generalizes the gw#591 boot-smoke pattern. P3 absorbs PR #353 (pgw#606/th#938) and PR #307 (pgw#583) as real hub-double dispatches (no mocking of gen_worker internals, unlike the original unit tests) - revert-verified red against lifecycle.py's pre-fix state, restored clean. P1/P2/P4/P5/P6/P7 absorb assertions from test_worker_grpc_e2e.py, test_shared_cas_root_multiwriter.py, and test_boot_smoke_gw591.py. Found + fixed in the harness itself: WorkerHarness must set the TENSORHUB_CACHE_DIR env var (gen_worker.models.cache_paths reads the process-wide get_settings() singleton, not the per-worker Settings instance) - passing tensorhub_cache_dir= to load_settings() alone silently no-ops, and every hub-double test defaults to a fresh per-call temp dir so downloaded bytes never leak between tests. 40 passed, 1 skipped (pgw#605 heartbeat: proto fields not generated yet) in ~33s local. Existing test files not touched or deleted (Phase 3).
…load metrics), plus ruff cleanup + a hub_double(file_base_url=) knob P9 needs P8 (tests/convert/test_p8_convert_publish_contract.py, hermetic fake_hub): dtype-mismatch cast honesty (pgw#589/th#901, condensed from test_publish_as_is_dtype.py), classifier corpus table (pgw#593), and an xfail(strict=True) test-first pin for pgw#566 (open bug: normalize_adapter _state_dict only passes unet_config to converters with a NAMED parameter - diffusers' real SDXL lora_state_dict takes **kwargs only, so the SGM block remap silently never runs). pgw#569's W8A8 verifier ULP gate is a documented skip - no factored-out comparator to test hermetically without a real w8a8 artifact round trip. P9 (tests/test_p9_result_upload_metrics.py): inline vs blob_ref by size alone over a real hub-double + real local media-upload dedup sink (no S3 part scripting, matching test_media_upload_owner.py's pattern); JobMetrics typed usage survives both wire forms. Needed hub_double(file_base_url=) so the sink URL rides the FIRST HelloAck instead of a racy post-connect mutate. ruff clean (unused imports, one E731 lambda->named helper). mypy: src/gen_worker clean (untouched); new tests/harness + test files carry the same untyped- fixture-param convention as the rest of tests/ (mypy only gates src/gen_worker per CI, tests/ was never in scope). Full new suite (P1-P9 + P8's own file): 50 passed, 1 skipped (pgw#605, proto fields not generated), 1 xfailed (pgw#566, fix open) in ~31s local.
test_slot_boot_precedence_th938.py, test_model_slot_identity_gate.py -> superseded by test_p3_slot_binding_precedence.py (same assertions, over a real hub-double boundary instead of monkeypatched Executor.store internals). test_shared_cas_root_multiwriter.py -> superseded by test_p4_cas_multiwriter_integrity.py (same 3 tests reproduced verbatim + 1 new row). tests/convert/test_publish_as_is_dtype.py -> superseded by tests/convert/test_p8_convert_publish_contract.py (same 3 core tests against the same fake_hub). Deletions only, no other changes. Full deletion manifest (all 166 files audited, reasoning for what's NOT deleted) is in the th#960/pgw#609 tracker checkpoint - the bulk of the pre-existing suite tests production surface (CLI, LoRA runtime, quantization lanes, VRAM/RAM admission, media/upload, Slot/discovery SDK, tests/convert producer suite) that no P-test covers, so it stays pending explicit direction on the larger absorb-or-accept-loss effort that would be needed to delete it safely.
…kstep with th#962)
… lockstep) - statvfs on the real mounts (CAS root = container tier; ismount-guarded fill source volume when attached; NFS tier reserved) — never a declared size - safely-reclaimable = ref-index bytes of DISK-tier refs inactive AND not in the desired set (the disk-GC LRU's eligible set); reuses ref-index bytes, no tree rescans - capacity_generation bumps only on a measured (64 MiB-quantized) shape change; rides Hello.state and every StateDelta; 30s periodic refresher is edge-suppressed
Manual dispatch + PR-into-master only (drafts/doc-only paths run nothing); drop the on-push trigger. Same gates (mypy gating, ruff non-gating per debt note, HTTP-timeout guard, llama-server, uv build); pytest targets the whole tests/ dir so P-suite additions land automatically.
This reverts commit c39efb69b82d967393bd3f3243ce328d954380f7.
…3/P5/ P6/P8/P9 + 2 new consolidated files, ahead of the bounded aggressive deletion sweep Bucket-level triage per the coordinator's ruling (Paul's directive: minimal greenfield set, ~92% removal). Absorbed only distinct bug-classes carrying a real pgw#/gw# incident number or a revert-catches-it story; untagged "tested thoroughly once" coverage goes without absorption (git history is the archive). P1 (+6): auth-rejection exit/retry-within-window, permanent-precondition fast-exit, hello-ack-deadline reconnect, not-leader redirect, hub-restart reconnect - absorbed from test_worker_grpc_e2e.py (#372) ahead of its deletion. P2 (+3): setup-failure FnUnavailable+recovery, host-RAM-failure wire ordering (th#807) from test_worker_grpc_e2e.py; corrupt-load quarantine+ retry-once (gw#408, J17 flood) from test_snapshot_corruption.py. P3 (+1): bare-ref prefetch binding resolution (#377) from test_executor_prefetch_binding.py. P5 (+1): heavy-dep stub contract (pgw#506) from test_discovery_heavy_deps.py. P6 (+1): SendQueue capacity-generation fencing from test_worker_grpc_e2e.py. P8 (+5): diffusers root-allinone skip (e2e J7 ENOSPC), standalone-component canonical-weight selection (gw#426), transformers sharded-index onnx/pickle exclusion, gguf explicit-quant refusal, non-safetensors-format refusal family - from test_classifier.py ahead of its deletion. P9 (+1): token-bound-owner routing (J19 run34) from test_media_upload_owner.py. New consolidated files (coordinator-authorized, one per lane with no P-test home): tests/test_quantization_lanes.py (W8A8/W4A4/FP8 contract + gw#409 promote-device-integrity, from test_w8a8.py/test_w4a4.py/ test_fp8_and_emergency_loading.py/test_promote_device_integrity.py) and tests/convert/test_convert_producer_contract.py (clone concurrency gw#442, disk preflight gw#462, download-skip bank th#592, publish resilience gw#462, from the tests/convert/ producer suite). Also reverted a premature local-only Phase 4 CI commit (c39efb6, from a prior session before an interruption) - never pushed, out of THIS issue's authorized scope ("do not touch CI yet"); preserved in history via revert, not discarded. Full new-suite gates: mypy clean, ruff clean, each new/changed file green in isolation. Deletion commit (the ~125 files these rows supersede) follows as a separate, deletions-only commit per the coordinator's mechanics.
…8 LoC) superseded or absorbed per the coordinator's ruling Every file here is either fully superseded by test_quantization_lanes.py / test_convert_producer_contract.py, absorbed as rows into P1/P2/P3/P5/P6/P8/ P9 (chaos b8274a1), or carries no incident pin (git history is the archive, per the coordinator's bound). Full manifest, bucket-by-bucket reasoning, and the dangling-import sweep are in the th#960/pgw#609 tracker Checkpoint 6, pushed before this commit. Deletions only, no other changes. Foreign-WIP clusters untouched: compile_cache/executor_adopt/fleet_cells (this issue's own claim-checkpoint note) and test_disk_telemetry_pgw610.py (landed on chaos by another lane during this session). Final state: 14 test files (3,293 LoC) + tests/harness/ (6 files) + conftest.py x2 + fake_hub.py, target was ~12-15 files per the coordinator.
…ycles skip __init__, same convention as _cancel_residency_reconcile)
…3 config user overrides are thread-local ContextVars Live disproof (0.40.5, B200, 2026-07-21): mint capture still packed 8 ASLR-keyed aotautograd entries and the store-served sibling still failed (warmups=1, cache_hits=0, cache_misses=8) — the 0.40.4 disable ran on the arming thread; the warmup compile thread saw the default True. The env var is read once at torch import (env_name_force), so post-import it is inert. Fix: entrypoint sets TORCHINDUCTOR_AUTOGRAD_CACHE=0 before any torch import (binds all threads + compile-worker subprocesses); _disable_aot_autograd_cache additionally mutates the installed config entry's env_value_force (process- global) for already-imported-torch embedders. Cross-thread revert-turns-red test added (fails on the 0.40.5 shape, passes with the fix). Dry evidence: fxgraph keys 8/8 identical across ALL five banked packs (0.40.1 x3 / 0.40.3 / 0.40.5); aotautograd level-2 keys 0/8 everywhere. samples/selfmint-proof/gw0405-run/.
…ngs AUTO-pick scale-free #fp8 locally gen_worker.models.ladder gains family_root (modelfamily.Root twin) and the conv-UNet w8a8 exclusion table; a maybe_rebind_family_fp8 fold (hub-parity AUTO pick over the resolve's sibling_flavors, sm_89+ only, bf16 stays the sub-floor default; pins untouched) runs in the local CLI resolve ahead of the GGUF fold over one shared resolve. WorkerResolvedRepo carries the resolve's model_family (slot-declared family as fallback); select_gguf no longer lets an AUTO-ineligible w8a8 row suppress the GGUF pick. Shared vectors: none in this repo; no format change.
… liveness contract (0.41.0) Hello declares heartbeat_interval_ms=30000; the pgw#610 disk-report task is promoted to the beat: it force-re-sends the full StateDelta every 30s from the asyncio event loop (the control loop that owes progress, never a detached thread), starts at startup() entry so boot hangs are covered, and keeps beating through drain until the stream closes. Edge-triggered deltas still flow as before; the beat is the same message re-sent unchanged. Contract §3 rewritten to the 3-layer model (lockstep tensorhub c03a531d). Tests: hello cadence, force-send bypasses edge suppression, beats flow while a startup coroutine is parked (the gw#612 shape), drain keeps beating until close.
…ing every beat HEARTBEAT_INTERVAL_MS 30000 -> 10000 (hub reaps at 6 misses, ~60s). The pgw#610 statvfs/ref-index scan keeps its 30s measurement cadence behind a TTL cache so the 10s beat never turns it into a hot loop; the report still rides every StateDelta. Contract §3 event-loop discipline clause added — already true in 0.41.0: setup/warmup/compile/residency/GC all run off-loop (executor._to_thread_complete / asyncio.to_thread).
…se; run-26 root cause corrected (0.42.0) Run 26's filed post-seal_publish hang is disproven: the worker completed setup and advertised readiness; the wedge was the hub's singular compile fence treating the 2-lane record's same-identity self-attested targets as ambiguous (tensorhub lockstep fix). Real worker defect fixed: a shared mint capture missing an unexercised mandatory sibling's graphs was still published as the family cell, bricking every adopting boot (gw#611 qwen hits=1/misses=1). finalize_self_mint packs only; the executor publishes iff every capture sharer proved in, else withholds typed+loud. New 'finalize' activity phase covers the post-proof tail to readiness.
… root cause: healthy-idle, hub fence starvation)
…965 layer-2 drills
…e fail-closed detail; cross-process portability repro Measured mechanism for th#954's release-bricking 0/0: bundled AOT cache serves a fresh-process adopt with fxgraph counters fully silent, so a HEALTHY serving cell read cache_hits=0/cache_misses=0 and fail-closed. inductor_counters now surfaces aot_cache_hit/aot_cache_miss and the guard wrapper counts AOT hits as serving evidence (AOT stays pinned off per gw#608; a config regression now degrades to a proven boot, not a brick). Fail-closed detail carries calls= to split orphaned-wrapper (calls=0) from counter-blind serving (calls>0). New tests: real mint->pack->fresh- subprocess adopt asserts >=1 FX hit; bundled-AOT 0/0 mechanism pin; guard credits AOT hits (red on revert). Hub lockstep: tensorhub chaos 9e7dca8e quarantines cells failing their own adopt proof.
Second family after SdxlDefaults: steps/guidance/guidance_2/max_guidance for the Wan 2.2 A14B MoE envelope. Registered under tensorhub's canonical architecture root "wan22" (not a Compile(family=) shape-grid string) so repo-metadata PUT validates against the matching tensorhub schema (companion wan22.schema.json on tensorhub chaos). Lets the wan-2.2 endpoint's T2V function migrate to the Slot model and give AnimeGen-T2V its own distilled recipe (8 steps, guidance=1.0, guidance_2=1.0) via repo metadata instead of branching on checkpoint identity.
…s publish) + on_hello_ack model-set-diff cancel (A) _run_synthesized_warmup coverage pass: compile-target objects still unexercised after the planned jobs get media VARIANTS of the same base payloads (warmup.media_variants: one optional image/audio field filled with a generated asset, nothing else drifts) — input-routed sibling lanes (qwen edit) mint into the shared capture, so the gw#612 gate publishes the union cell instead of withholding forever; adopts prove the sibling lane instead of arming unproven. (B) th#961 defense in depth: a HelloAck whose semantic model set (resolutions+disk_refs+snapshots+hot) matches the running reconcile's target no longer cancels it (in-flight self_mint_compile survives benign plan rewrites); changed sets cancel as before.
…seal_publish hang pgw#610/th#962's disk_usage_report() ran statvfs()/stat() synchronously inline in Lifecycle._state_delta(), which many hot paths call directly with no event-loop offload — including right after a self-mint's seal_publish. The provider-attached VOLUME fill-source is a network-backed mount that can stall for minutes under load (exactly what a self-mint's weight download + cell pack produce). A stalled statvfs() there freezes the whole event loop: every StateDelta, RunJob dispatch, and even the th#965 heartbeat (same loop) stops until the syscall returns. ModelStore.disk_usage_report() now only reads a cache; the actual measurement (_measure_disk_usage_report) runs exclusively via refresh_disk_usage_report(), off-loop through asyncio.to_thread. Lifecycle kicks that refresh as a fire-and-forget background task (never awaited inline), gated to the existing ~30s TTL, so a stalled mount delays only its own telemetry freshness — never any send, dispatch, or heartbeat. Root-caused via git diff v0.40.5..v0.40.7 (executor.py/lifecycle.py) plus gw#612's hub-log forensics for the qwen case: gw#612 itself was hub-side (dispatch-fence starve, corrected root cause, no worker hang) and its worker-side companion fix addresses a different defect (poisoned family-cell publish on the NEXT boot). This is a distinct, genuinely-new gen-worker regression introduced by a70e7c9 (wave-2 telemetry), reproduced locally with no GPU via a stalled-statvfs monkeypatch. Tests: tests/test_disk_telemetry_pgw610.py (new test_disk_usage_report_never_measures_only_reads_the_cache, test_state_delta_never_blocks_event_loop_on_a_stalled_mount; existing tests updated to the cache/refresh split) + tests/test_heartbeat_gw613.py (new test_beats_flow_while_disk_measurement_is_stalled; existing TTL test updated to the fire-and-forget refresh). All revert-turns-red verified. Full suite 458 passed/5 skipped; ruff+mypy clean.
…y; keep master's Phase-4 ci.yml)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes the remaining reviewed chaos work; first serving-safe version after the
0.40.7 post-seal_publish hang.
heartbeat) on a stalled volume-mount statvfs(). _state_delta() now reads only
ModelStore's cache; measurement is a fire-and-forget asyncio.to_thread refresh
gated to the report TTL.
(SELF_MINT_PUBLISH_WITHHELD), finalize activity phase; run-26 root cause
corrected.
cells publish) + on_hello_ack semantic model-set-diff cancel (th#961 defense).
detail + real-subprocess portability repro.
on PyPI for inference-endpoints' wan-2.2 slot migration.
wedge probe example.
Version: 0.42.0 (matches chaos's own bump; 0.41.0 already tagged+published by
PR#362 — 0.40.8 would sort below the published latest). master's Phase-4 ci.yml
kept (chaos's copy was a reverted local commit, not intent).
Local (py3.12, torch 2.13, locked venv): 480 passed, 5 skipped, 1 xfailed —
identical on chaos tip and this branch; mypy + ruff clean.
Note: gw#614 carries a documented GPU-verification caveat (1:1-bucket synthesized
edit image) — tracked in its issue; does not block the wheel.
🤖 Generated with Claude Code