th#1055: desired-hot warm works on slot-only endpoints; refusals emit model events (0.50.2)#379
Merged
Merged
Conversation
… model events (0.50.1) ensure_desired_instance required the hot instance's binding set to equal spec.models, but deploy-bound Slots (ie#524/th#980) have no code default so spec.models is empty fleet-wide — every hub hot intent (gw#587 self-mint prewarm, th#912 seeding, #567 cell reload) failed a ValidationError that _reconcile_pass swallowed as one pod-local warning: no warmup, no self-mint, the th#868 w8a8 fence never opened, cells never armed. - validate against the declared slot set; code defaults may fill their own - declared-space hot bindings remap through the HelloAck precision picks - every desired-instance failure emits MODEL_STATE_FAILED for the instance refs (the silent stall is now impossible)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause of the fleet-wide silent warmup stall (th#1055 / th#1044 item 4 / th#1049 ltx deadlock / th#1059 worker silence):
ensure_desired_instancerequired the hot instance's binding set to EQUALspec.models. Deploy-bound Slots (ie#524/th#980) carry no code default, sospec.modelsis EMPTY on every fleet endpoint (qwen, sdxl, ltx, z-image, ernie, klein, anima, hidream) — every hub hot intent (gw#587 self-mint prewarm, th#912 slot-default seeding, #567 compile-cell reload) raisedValidationError("must bind exactly []"), swallowed by_reconcile_passas one pod-local warning. No warmup, no self-mint, zero events, 0% VRAM; the th#868 mandatory-w8a8 fence never opened and precompiled cells never armed.Live evidence (master hub log, worker d7bfd1ea / pod rjvp3srr8s7zsh): hot intent
HelloAck (desired_generation=3 disk=2 hot=1)sent 20:09:02Z one second after the last fp8 download hit disk, then 95 minutes of total worker silence untilcold_idle_never_dispatched.Fix:
spec.models|spec.slots); missing slots acceptable only when a code default fills them (dispatch's own fallback semantics)Tests: tests/test_desired_hot_th1055.py (4 tests, revert-turns-red verified against the pre-fix executor). Full suite green locally (663 passed; one known-flaky zero-cpu watchdog test failed once under box load, passes on rerun).
🤖 Generated with Claude Code