Repository navigation
Replies: 2 comments
|
@CODEG333K Please upgrade the code. The prefix allocation fix landed in #1056, included in v0.9.3. Optional NAR windowing reduces memory further, with a context/quality trade-off. Q6_K package, fixed 8,192 semantic frames (~327.7 seconds), two denoising steps, and the same F16 VAE on RTX 5090.
Also in my tests, at matched lengths, Q6 stayed about 360 MiB above Q4. |
|
Upgraded to v0.9.3 (
You were right and I was wrong. The widening was #1056's prefix path: the 1,074 MiB my two-step probe found appearing only once generation started was the host KV export/staging, and on So nothing here needs Thanks for the quick, precise answer — the engine was easy to re-measure because the API reports what it actually loaded. |
Uh oh!
There was an error while loading. Please reload this page.
YuE2: a Q6_K pack costs as much VRAM as Q8_0 — the k-quant weights are widened during generation
Hi @0xShug0 — thanks for audio.cpp. I run an edited Remiqora build in my Docker Engine in Windows, a local-first music studio (Electron shell, FastAPI backend) that wraps your engine with a UI, a track/project layer and a small orchestrator so YuE2, ACE-Step and the stem/MIDI tools can share one GPU. I compile
audiocpp_serverfrom source for the Linux/CUDA container and run it on an 8 GB laptop card, so the memory ceiling is something I hit daily rather than occasionally.Which is why I repacked a Q6_K conversion of YuE2-3B for the engine and measured it against the stock Q4_0/Q8_0 packs. The result surprised me enough to write it up: Q6_K is the one precision in the family that saves no memory at all. Details and the probe that localises the extra allocation below — I have not profiled the engine, so treat the cause as a reasoned guess and the numbers as measured.
Environment
881be1d69f5c4da869ffbd75e1ac55d954250b89(Lower YuE2 context defaults, Lower YuE2 context defaults #767), built withscripts/build_linux.sh --backend cuda --cuda-arch 89 --target audiocpp_server, run in server mode (--ui-management --backend cuda), driven throughPOST /v1/tasks/run.yue2(YuE2-3B +yue2-vae-f16.gguf),cot=full,cfg_scale=1.2,num_inference_steps=50, default semantic budget (9,000), same lyrics and seed for every run. VRAM sampled every 5 s withnvidia-smi, peak = max of the samples.What I checked first, so this is not a duplicate
q6_k: no matches. ForVRAM: eight threads, none about k-quants. Issues search forq6_k: 14 hits, all about other families' Q6_K/Q4_K packaging (fish_audio, confucius4_r2t2, lfm2_audio), none about YuE2 memory.docs/gguf.mddocumentsq2_k…q6_kas "Lower-bit quantized formats. Treat as experimental per model and backend", and the Q8 packaging notes already describe keeping norms, biases and scalar tensors out of the quantized type — but nothing there says what a k-quant costs in VRAM at runtime.tonight-awake): BF16+F32 12,535 MiB / Q8_0+F16 8,867 MiB / Q4_0+F16 7,755 MiB. Those already show the Q8_0→Q4_0 step saving ~1.1 GB for a ~1.5 GB file difference; there is no Q6_K figure, which is the gap this fills.yue2-3b-ios-q4_0.ggufin the public YuE2 GGUF repo is the lightest thing I measured — 5,958 MiB on the 8 GB card. I assume it belongs to the same low-memory direction as the 1.7 GB iPhone figure you mention in Add support for Yue2 #691; if so, that path plus a q6_k variant is the combination worth having.What happens
Q6_K is the only precision in the YuE2 package family that buys nothing. A Q6_K pack that is 1,076 MiB smaller than the Q8_0 pack reaches the same peak, and it is 1,624 MiB worse than Q4_0 despite being only 449 MiB larger on disk:
yue2-3b-q4_0.ggufyue2-3b-q4_0.gguf(ios layout, embed in q8_0)yue2-3b-q6_k.ggufyue2-3b-q8_0.ggufThe last row was recorded on the same engine build but with
--max-loaded-models/--min-free-memory-mbstill in the launch command; dropping those flags moved the Q4_0 run from 7,644 to 6,298 MiB, so the Q8_0 row is only indicative and not comparable. The first three rows were all measured on the same launch configuration, and the q6_k pack runs to completion (audio + ABC artifact) — it just does so with 35 MiB to spare.The probe that localises it
A 2-step
cot=offrun with the same seed is short enough that the peak is essentially the resident weights plus a small workspace:yue2-3b-q4_0.ggufyue2-3b-q6_k.ggufyue2-3b-q8_0.ggufQ6_K is 550 MiB above Q4_0 when nothing is generating (that part tracks the file size), but 1,624 MiB above it on a full song. The ~1,074 MiB difference between those two deltas is allocated while generating, not held as weights: it is not a bigger resident model, it is a larger per-run working set. Q8_0's probe is only 326 MiB above Q6_K's, which is why both formats end up at the same ceiling — the ceiling belongs to the generation, not to the file.
Likely cause
Widening. A q6_k block is 6.5625 bits/weight; there appears to be no kernel path that multiplies q6_k blocks in place for these shapes, so the weights are materialised as f16 tiles (2.44× the q6_k footprint) for the matmul, and the graph arena stays sized for that worst case. q4_0's frozen blocks are consumed as they are, which is why q4_0 is the only format that actually saves memory here. I have not profiled the allocation, so this is inference from the deltas rather than a proven call stack.
What would fix it
cudaMemcpyAsyncfailing, or the well-known 100 % util / ~20–30 W stall.Two smaller things noticed while repacking a Q6_K conversion for this engine
audiocpp.tensor_nameswith the tensor table positionally. A pack that is correct by name but in a different order fails withtensor byte size mismatch for model_weights/model.embed_tokens.weight, naming the embedding while every tensor after the first difference is shifted. Sorting the table into the reference pack's order fixes it.model_weights/model.embed_tokens.weight(q8_0) andtime_embedder.mlp.*.weight(f16) are rejected as q6_k with the same message. Keeping the stock payload for those three and using q6_k everywhere else is what makes the pack loadable — worth documenting next to the Q6_K package, since a naive conversion hits both.Repro
Server mode, one request per pack after
POST /v1/models/loadwithsession_options: {"yue2.model_gguf": "<file>"}, body{"model":"yue2","request":{"lyrics":"…","seed":838222690,"options":{"style":"…","cot":"full","cfg_scale":1.2,"num_inference_steps":50}}}, withnvidia-smi --query-gpu=memory.used --format=csv,noheaderevery 5 s. The repack tool used for the Q6_K pack is a separate script (repack_audiocpp_gguf.py); it only rewrites the container, it does not touch tensor bytes.Attachment:
repack_audiocpp_gguf.py— the tool that produced the Q6_K container, in case the pack itself is the interesting part. It reads a plain GGUF and rewrites it into the packed layout this engine loads (reference metadata, namespaced tensor names, the borrowedlatent_pos_embed.pe), copying the tensor payload byte for byte. It was validated by re-emittingyue2-3b-q4_0.gguf1:1, which came out byte-identical (same SHA-256), and the q6_k pack it writes is 628 tensors: 394 q6_k, 229 f32, 3 f16, 1 bf16, 1 q8_0. If you would rather I regenerate it withaudiocpp_gguf --type q6_kand re-run the numbers against that output, say so and I will — the measurements above do not depend on how the container was produced.Happy to run any build you want to try, or to re-measure on a bigger card if that helps separate the widening from the arena sizing. Thanks again for keeping the engine scriptable enough that a report like this is possible at all.
Sincerely,
CODEG333K
repack_audiocpp_gguf.py
All reactions