Skip to content

gpu/vk: prototype sparse guest-memory imports on Linux - #38

Draft
biosed wants to merge 17 commits into
pixl27:mainfrom
biosed:codex/sparse-guest-prototype
Draft

biosed wants to merge 17 commits into
pixl27:mainfrom
biosed:codex/sparse-guest-prototype

Conversation

@biosed

@biosed biosed commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Adds an opt-in RX_VK_SPARSE_GUEST Linux path using strict non-resident sparse-buffer semantics, memfd extent discovery, UDMABUF_CREATE_LIST imports, and host gap release. The normal complete-import path remains the fallback.

Tests:

  • python3 nx2/test_region_gaps.py
  • python3 nx2/test_sparse_guest_ranges.py
  • python3 nx2/test_import_windows.py
  • git diff --check
  • full Fedora BC-250 build
  • guarded same-route hardware run with correct 45-second load-menu capture

Current hardware finding: this reduces early committed memory and preserves rendering, but blocks remain bound for the pool lifetime. A 111-second Kokiri load still reached the 4 GiB host safety floor, 2 GiB cgroup swap, and roughly 4.25 GB GTT, so this is intentionally a draft prototype rather than a claimed memory fix. DirectMesh submission churn is being investigated separately.

@biosed

biosed commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Sparse guest release milestone pushed as biosed/ProjectReLink@3c64b03.

This records exact sparse-allocation runs, batches safe VK_NULL_HANDLE unbinds after a queue drain, frees whole dead imported allocations, clears residency bits, and lets the existing host gap release punch pages afterward. The primary bookkeeping allocation is bounded to one sparse block.

Gates: device-free sparse/range/import suites pass locally and with the BC-250 clang 22 toolchain; full renderer compile/link passes. Candidate relink_vk.so SHA-256: aa6aab9e1ac98f3fe9864429ee126c04e30014e1bc40e21b5c66da85718286c0. Guarded hardware route is next; PR remains draft pending those results.

@biosed

biosed commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Guarded BC-250 hardware route for 3c64b03 completed under 5.25 GiB MemoryHigh / 7.5 GiB MemoryMax / 2 GiB swap / 5 minute cap. No crash, Vulkan error, unbind failure, or host-memory guard trip.

The route did not reach an allocation eligible for this change: the game had freed only 26 MiB of address regions, and host telemetry reported 0 MiB of those held under GPU imports. Accordingly there were 0 sparse releases (and 0 failed releases). This corrects the current bottleneck diagnosis: late GTT flattened around 3.779 GiB while live scene construction stalled under reclaim and virtual-core starvation; stale sparse imports are not the source of this particular peak.

Exact extrema: cgroup 5,707,792,384 B; swap 2,147,483,648 B; GTT 3,820,539,904 B; shmem 5,073,768,448 B; minimum MemAvailable 4,443,680 KiB; 57 C; PSI some +112.1 s. Evidence hashes: log 3c52124906cae1ecbfb1844e897a72971c744b600fa12a80daf27be74ffbcdc6, metrics 47bfbc66a721a5b11fe1f4da010ad3375812c5f1851c0e91587242af7ec22d2c.

PR remains draft: the mechanism compiled and behaved conservatively, but its intended release path still needs a route or isolated Vulkan harness that actually makes an imported sparse allocation dead.

@biosed

biosed commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up checkpoint 1504a5f restores virtual-core preemption inside the native name-search optimization.

The native helper now mirrors every replaced lifted A64_POLL: a relaxed Ctx.poll check at entry and at the binary-search/long-strcmp backedges, clear-before-yield on the slow path. This targets the guarded BC-250 run where priority-26 PreActorModelCompose repeatedly escaped after 2 s while priority-27 OptimizeChangeBanc held cores marked asked; the native helper had bypassed the lifted polls.

Verified:

  • device-free test_names_native.py locally and with BC-250 Clang 22
  • sparse range, region-gap and import-window harnesses locally and on BC-250
  • full BC-250 v0.7.5 shadow rebuild passed (names_native.c new; 0 compile failures)
  • built object disassembly contains the poll checks at entry and the native backedge

A same-profile guarded hardware A/B is next; no performance claim yet.

@biosed

biosed commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Hardware-verified follow-up f267303 releases the guest virtual core across six synchronous renderer callbacks that can wait on replay/GPU work or renderer mutexes (texture/sampler registration, texture/program release, CPU read/write texels).

Same-profile BC-250 guarded comparison, with commit 1504a5f already present in both:

poll-only host + renderer-call coverage
wall until 4 GiB host floor 117 s 127 s
virtual-core escapes 7 0
OptimizeChangeBanc … asked escapes 1 0
last native-search count 28.9 M 29.0 M
furthest slow-upload id #471 #513
PSI-some delta 2.57 s 2.36 s
max swap 2.00 GiB 1.83 GiB
max GTT 3.85 GiB 4.20 GiB (farther route)

Both runs stopped by the independent MemAvailable < 4096 MiB guard; no crash/temperature stop. Full BC-250 build passed with rx_backend.c new and 0 compiler failures. Device-free scheduler/sparse/range/import tests remain green. The next boundary is memory growth, not virtual-core starvation.

@biosed

biosed commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

BC250 hardware milestone at 0087623: integrated/configured texture allowance now drives ASTC BC3 and staging retention. Pinned BC250 toolchain passed test_texture_memory_policy plus the four sparse/scheduler fixtures; full candidate build completed with 0 compile failures. Same guarded profile/limits reached upload #543 and 41M native-name searches versus #513 and 29M before, with 0 vcore escapes and 0 Optimize asked. Peak GTT was 4,459,634,688 B versus 4,509,126,656 B. The unchanged 4 GiB host-availability guard still stopped at ~2m because shared-memory/commit pressure remains the next boundary; Plasma/KDE Connect were restored.

@biosed

biosed commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

BC250 sparse-memory milestone: 93db628 splits aligned sparse guest runs across the kernel udmabuf byte/extent limits instead of falling back to dense imports; 6ec1614 zeros only allocated memfd extents when released guest ranges overlap GPU imports, preserving untouched holes. Six device-free fixtures pass on both local and pinned BC250 toolchains; full candidate build completed with 0 failures. Identical guarded hardware run: 4 oversized pools split successfully, 0 oversize failures, 0 dense fallbacks, 0 vcore escapes, 0 Optimize asked, upload #726 / 43.4M searches versus #543 / 41M before. Peak anonymous RSS fell 1,951,028 -> 739,832 KiB, swap 2,078,121,984 -> 1,510,477,824 B, and cgroup PSI 2.50 -> 1.24 s. Desktop restored after the 4 GiB host-floor guard.

Use SEEK_DATA on the native Linux guest memfd before scanning a large texture for zeroes. A wholly unallocated range already reads as zero, so clear the Vulkan image without instantiating every shmem page. Keep the helper in the host sparse-file boundary and make it an explicit renderer build dependency.

BC-250 run 19 recognized the 288 MiB and 32 MiB shadow targets in 0.3 ms and 0.1 ms, versus about 1.1 s plus 320 MiB of resident shmem in run 18.
@biosed

biosed commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

BC-250 guarded hardware A/B for a3333b2:

  • Before (run20): movie path collapsed to 0.2 FPS; guest workers repeatedly waited about 2 seconds for the virtual core held by synchronous libavcodec/dav1d decode.
  • After (run21): movie and rendered Kokiri save/load menu sustained roughly 57-60 FPS; post-startup vcore waits were generally milliseconds/tens of milliseconds, with zero escaped cores.
  • Memory during that phase held around 5 GiB with negligible swap. The same run later exposed a separate 3D area-load bottleneck and was stopped by the host-memory guard; this does not invalidate the video A/B.
  • Full remote standalone build succeeded; git diff --check passed. Gemini independently reviewed lock ordering and all returns, confirming hos_vcore_back() happens only after releasing the decoder lock.

Evidence remains outside git under the BC-250 run21 logs/screenshots.

@biosed

biosed commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Milestone 8ba548e keeps Linux dav1d planar 4:2:0 frames on the existing NV12 path instead of allocating/converting an unused 8.3 MB RGBA image per 1080p frame.

BC-250 guarded A/B: title movie remained 57–60 FPS; HOS_VIDEO_LOG confirmed no queued RGBA buffers; at acquire 60 the movie was ~0.15 s behind (previous host-video-vcore run ~0.73 s), with roughly 300 MB less memory at the same route point. The later 3D-load collapse is independent and has now been isolated to sparse_guest_bind/memory_begin_submission. Full standalone Linux build succeeded.

@biosed

biosed commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Milestone 6188cff bounds Linux sparse-guest refresh work by Vulkan sparse block: it skips already-bound blocks, skips remaining shmem extents after the first data byte in a 64 KiB block, and batches all sorted live guest-pool intersections into one scan/bind sequence per imported pool. Added isolated coverage for range merging, unbound selection, and scan advancement.

Validation: sparse range/file fixtures PASS; full Vulkan-only Linux build compiled 188 files with 0 failures. BC-250 guarded run29 with renderer hash ff731242… held 57.7–60 FPS for ~45 s and entered the cold 3D transition at 56.1 → 46.7 → 17.9 FPS, versus prior 3 → 0.4/0.2 FPS, with no >=20 ms sparse scan reports. The guard then stopped at the 3.5 GiB host-memory floor (cgroup 7.32 GB / guest shmem 5.75 GB); CPU scanning is fixed, but the scene-load memory peak remains the next blocker.

@biosed

biosed commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor Author

Milestone update: 70f8fa1 makes sparse imports bind only their primary block initially, then binds the exact direct-source range in gpu_buffer() before use. Descriptor-backed gpu_buffer_bind() already follows the exact-range path.

This removes a measured 2.6 s SEEK_DATA scan over a 26 MiB import chunk for a small texture. On the guarded BC-250 route, new imports were generally 2–8 ms and the previous full-chunk spike disappeared; correct Kokiri rendering was preserved. Fixtures passed locally and in the BC-250 container, Vulkan-only build passed 188/188, and git diff --check passed. This improves streaming latency; it did not change the steady DirectMesh CPU bottleneck.

@Yel1oww Yel1oww mentioned this pull request Nov 15, 2026
@pixl27

pixl27 commented Nov 16, 2026

Copy link
Copy Markdown
Owner

Thanks a lot for this, and for the detailed BC-250 notes: they're useful well beyond Linux.

I've read through it. It can't go in as one piece yet, and here is what would let the parts move.

Scope. This started as an opt-in Linux sparse-import prototype, but it now also changes things every Windows player gets: the storage-buffer rule for fragment stages, giving up the virtual core around six renderer calls and the video decoder, the integrated-GPU texture budget, and names_native.c. The sparse part is nicely fenced (__linux__ + RX_VK_SPARSE_GUEST); the rest isn't. Could you split it? Roughly:

  1. sparse imports (still a draft, as you say);
  2. the win32_posix tid / priority / affinity fixes: small and Linux-only, I'd be happy to take those soon;
  3. the virtual-core release around renderer calls and video decode;
  4. the native cbuf pointer rule;
  5. the integrated-GPU texture budget.

Windows impact. Anything under gpu/vk/eden changes the shader cache's stamp, so the cbuf rule means every player redoes the long first-start shader preparation, and we'd want before/after captures on Windows for the fragment-stage part. For the renderer calls: I checked the call sites in nvn_native.c and none holds a host lock across them, so no deadlock worry, but a frame thread giving its core up on every texture or sampler registration may wait for one when the cores are busy. That needs an A/B on Windows first.

names_native.c. You're right that the native loops dropped the lifted code's polls. That was our miss, and zstd_native.c had the same gap. Both are fixed on our side for the next release (checked against the lifted code: 62 million name-search steps and 40,000 zstd sequence loops, no difference), so this hunk will conflict; I'd drop it from the PR. One note on the diagnosis: each native search is bounded (a binary search, at most 32 steps over short strings, microseconds), and the native code never clears the poll flag, so a kick is only delayed until the lifted code's next back edge after the call returns. I don't think it can hold a core for 2 s on its own, and your own table points the same way (7 escapes with the polls only, 0 once the renderer calls released the core). The batching of the search counter is a good catch, I'll look at it.

Small things. The suyu CMakeLists hunk lists ir_opt/native_cbuf_pointer_policy.h, which only exists under gpu/vk/eden (nothing builds that list, so I'd drop the hunk). In nn_video.c, planar frames now get a malloc'd NV12 buffer, so an odd width or height leaves the last chroma column or row uninitialised. Harmless at 1080p, but keeping calloc there costs nothing.

Happy to keep this open as a draft while the sparse work goes on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants