gw#627 train: Conv2d runtime-LoRA branch (0.51.0) + th#1063 fill-source boot guard (0.50.3)#380
Merged
Merged
Conversation
…0.50.3) A RunPod pod without TENSORHUB_FILL_SOURCE_DIR (or with a non-mount path) pulls everything from R2 with write-through silently off — the defect that hid the fleet-wide unattached-volume regression. Boot now logs fill_source_disabled (reason=unset|not_a_mount) on datacenter pods and fill_source_enabled when the volume tier is live.
….51.0) Curated sdxl distill adapters (Lightning/DMD2) carry 49 conv LoRA pairs; the gw#547/558 branch was Linear-only, leaving raw peft as the only route on the w8a8 lane — fatal on _Fp8ScaledLinear (8/8 live turbo failures, th#1037 addendum). Plain nn.Conv2d denoiser modules are now branch-capable: canonical zeroed conv branches in the same rank bucket, staged-copy swap, eager instance-forward wrap conv1x1(conv(x,A),B).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release train (chaos -> master): two lanes.
gw#627 (0.51.0): Conv2d additive-branch support in the runtime LoRA overlay
Live P1: sdxl generate-turbo was 100% fatal on the fp8 lane (8/8 on L4, th#1037
addendum, request 16205a14) — the endpoint's curated Lightning/DMD2 adapters
went through raw diffusers/peft
load_lora_weights, and peft rejects_Fp8ScaledLinear(pertensor). The sanctioned route (gw#547/558 branch viaAdapterResidency) could not carry them either: each adapter ships 49 CONV LoRA
pairs and the branch was Linear-only. This adds plain
nn.Conv2ddenoisermodules as branch targets: canonical zeroed conv branches in the same rank
bucket (A [bucket,in,kh,kw] at base stride/padding, B [out,bucket,1,1]),
same staged-copy swap, eager instance-forward wrap
conv1x1(conv(x,A),B).Convs are never quantized, so every lane takes the wrap path; cell lane naming
(
w8a8-lora<bucket>) unchanged. e2e-style CPU test over the real codepath:real tiny UNet2DConditionModel + real
_Fp8ScaledLinearpertensor swaps +live key grammar (kohya-flat, conv 4-d pairs) through
AdapterResidency.activate/deactivate under the compiled marker.
Endpoint half (sdxl overlay rewire, lora_bucket, warmup) follows in
inference-endpoints pinned to 0.51.0.
th#1063 (0.50.3): loud boot log when a datacenter pod has no CAS fill source
Chaos 77c2e11 (Paul) — rides this train per the th#1063 tracker note.
Local: 670 passed / 5 skipped / 1 xfailed; mypy clean; ruff clean.
🤖 Generated with Claude Code