Repository navigation
fix(apps): keep transitive torch out of GPU artifacts - #623
Open
justinwlin wants to merge 7 commits into
Open
justinwlin wants to merge 7 commits into
justinwlin wants to merge 7 commits into
Conversation
This was referenced Oct 17, 2026
Open
Packages that depend on torch (peft, accelerate, sentence-transformers) installed it into the artifact even on builtin GPU images, because only top-level requirements were excluded. The vendored PyPI torch shadowed the image's build and failed to load its CUDA libraries, so workers crash-looped. runpod-flash stripped these at archive time. After vendoring, remove every installed excluded distribution, plus the wheels only the torch family depends on (nvidia/cuda, sympy, networkx), using each wheel's RECORD and Requires-Dist. Fails closed to name-only removal when metadata is missing or a requirement has no name. Script paths pip --target records as ../../bin are mapped onto the target. Auto-exclusion now applies when any resource is GPU on a builtin image and none uses a custom image, matching runpod-flash for mixed CPU+GPU apps. A vendored PyPI torch cannot import on the builtin CPU images either, so CPU resources lose nothing.
Workers pip-install manifest excludedPackages their image lacks at cold start. In a mixed CPU+GPU app the GPU images already have the torch family, so only CPU workers act on the list, and a CPU stage that never uses torch tried to install CUDA torch onto its default 10 GB disk and failed to boot. In mixed apps, keep a torch-family name in the manifest only when the CPU resources' dependency closure (their decorator dependencies plus requirements.txt) reaches it, and warn that those workers install it at cold start. Unreadable metadata keeps every name, as before.
…or it transformers lists torch under its [torch] extra. the cpu-needs closure followed every Requires-Dist line, so a mixed app whose cpu stage used plain transformers recorded torch and its workers installed it at cold start.
justinwlin
force-pushed
the
fix/build-prune-transitive-torch
branch
from
October 20, 2026 15:48
9005f22 to
1889c3f
Compare
justinwlin
marked this pull request as ready for review
October 20, 2026 15:48
KAJdev
approved these changes
Oct 20, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flash → SDK 2.0 fix stack
PRs 1–9 are independent PRs off
main, and 10 is stacked on 9. Merge in this order. Hover a link for its status.mainpeft/accelerategets bundledrp flash devleaks endpoints when stopped by a signalvol.pathsays "not mounted" on deployed workersrunpod2.0.0rc2 with 1–6GPU apps whose dependencies pull in torch (
peft,accelerate,sentence-transformers) crash-loop afterrp flash deploy. runpod-flash didn't have this problem, so Flash apps migrating to 2.0 break.Cause
split_exclusionsonly skips torch-family packages listed as top-level requirements. pip still installs them as dependencies, and the vendored PyPI torch shadows the GPU image's build. A torch built off Linux has no NVIDIA libraries, so it fails to load (libcudart.so.13/libcublasLt.so not found). runpod-flash removes these at archive time.Change
torchrun). This uses each wheel'sRECORDandRequires-Dist, the same approach as fix(build): keep torch's dependency tree out of tarballs flash#390. Without complete metadata, or when a requirement has no name (URL, path), only the excluded packages themselves are removed.excludedPackagesthat their image lacks. In mixed apps, torch is listed only when the CPU resources' dependencies need it (a dependency behind an extra counts only when that extra is requested), and the build warns that those workers will install it at cold start. Otherwise a CPU stage that never uses torch tries to install CUDA torch onto its 10 GB disk and fails to boot.Testing
dependencies=["peft"], 4090 poolmaintorch/stripped at archive timelibcudart.so.13: cannot open shared object file2.9.1+cu128, CUDA available2.9.1+cu128, CUDA availabletransformers+ GPUpeft(--build-only)excludedPackages["torch"], plus "cpu resources need torch" warningHow we tested and compared
torch.__version__,torch.__file__andtorch.cuda.is_available(). It was deployed with runpod-flash 1.21, with SDKmain, and with this branch, all on the same GPU pool. We compared artifact size, bundled packages, and which torch the worker loaded./usr/local/lib/python3.12/dist-packages/torch. This PR's worker loaded/usr/local/lib/python3.12/site-packages/torch/__init__.py. Both are the image's build.humanize) plus a GPU queue (torch,peft): both returned results. Before the manifest change, the CPU worker failed bootstrapverifywithNo space left on devicewhile installing torch.transformers5.20.2 liststorch>=2.6; extra == "torch". The--build-onlycomparison above used real pip resolution and showed the first version recording torch for a CPU stage that never imports it.docker run python:3.12-slimwith a macOS-vendored PyPI torch onPYTHONPATHfails withlibcublasLt.so.*[0-9] not found.nvidia/dirs,../../binscript paths, missingRECORD/METADATA, unnamed requirements, user--excludedependencies, custom images) and mixed-app manifests (CPU need, direct GPU torch, an unrequested extra, a requested extra, and a marker branch that applies without the extra).pytest tests: 1496 passed, coverage 91.94%.Review (Codex, two rounds)
Requires-Distline, so plaintransformersrecorded torchextra == "torch" or python_version >= "3.12"was treated as gatedorbranch names an extra.RECORDfalls back to removing onlytorch/RECORDfor the installs this build makes, and the fallback exists only to fail safe.