Repository navigation
docs(flash): move the flash lane to the Runpod SDK 2.0 (rp flash) - #89
Open
justinwlin wants to merge 9 commits into
Open
justinwlin wants to merge 9 commits into
justinwlin wants to merge 9 commits into
Conversation
This was referenced Oct 19, 2026
Open
Open
justinwlin
force-pushed
the
docs/flash-to-runpod-sdk-2
branch
from
October 20, 2026 15:35
c741380 to
20150f5
Compare
justinwlin
marked this pull request as ready for review
October 20, 2026 15:48
justinwlin
force-pushed
the
docs/flash-to-runpod-sdk-2
branch
from
October 20, 2026 17:59
20150f5 to
bcb59eb
Compare
Flash ships in runpod 2.0 as SDK apps (runpod.App, @app.queue/@app.api/@app.task) with the rp CLI, and runpod-flash is deprecated on October 29, 2026. - flash skill: SDK-first SKILL.md, setup/CLI, API and patterns references checked against runpod 2.0.0rc1 (--help, source, sdk-runtime) - new references: migrate-from-runpod-flash.md, runpod-flash-legacy.md - examples/: five single-file apps, each passing rp flash deploy --build-only - evals rewritten for the SDK, plus a migration eval; dev-loop fixture now fails on a missing dependency (the body-only NameError no longer exists) - flash concept rules rewritten with public docs/source evidence; citing rules in flashboot, job, model-store, runpodctl, serverless-handler and volume-cache updated - router, READMEs, getting-started and endpoint/pod references use rp - golden paths 03-B, 07, 08 ported to the SDK and marked ported (build-only checked; the live runs used runpod-flash)
A non-interactive shell starts & jobs with SIGINT ignored, and SIGTERM exits rp flash dev without deleting its temporary endpoints. Verified live: the python exec wrapper restores SIGINT and the session reports its endpoints deleted.
- examples: queue_gpu, api_model_service and cpu_gpu_pipeline pass rp flash dev --once live; cpu_gpu_pipeline now casts to float32 because whole-number floats cross the job boundary as ints - golden path 03-B live-verified on the SDK; pin av<19 (PyAV 19 dropped the metadata_errors argument faster-whisper passes) - gotcha 14: dev-session runtime images predate runpod.NetworkVolume, so volume apps (train_then_serve, golden paths 07 and 08) fail in rp flash dev
Deployed SDK workers attach the network volume, but on the current runtime images vol.path raises 'is not mounted in the current execution'. Reading the literal mount path works: on 2026-10-16 one deployed endpoint wrote to a volume and another read the exact contents back. Golden paths 07 and 08 and train_then_serve.py now read literal paths and run via rp flash deploy.
…ment deploy crash loop - golden path 08 serve step verified via rp flash deploy (TinyLlama + LoRA from a network volume); call_deployed.py verified against deployed apps - gotcha 13: torch-dependent packages (peft, accelerate, sentence-transformers) vendor a CUDA 13 torch that replaces the image's and crash-loops GPU workers (libcudart.so.13); install them with pip --no-deps at runtime (08 shows how) - patterns.md drops accelerate from GPU examples for the same reason - api.md: rp flash deploy maps GPU names to whole pools; pin after deploy - concept rules flash.transitive-torch-breaks-gpu-deploy and flash.deploy-gpu-names-map-to-pools with live-probe evidence
The volume version could not run in a dev session on the current runtime images. The task now returns its weights as JSON to the queue endpoint (the volume handoff is covered by golden path 07). Verified 2026-10-16 with rp flash dev --once: the task Pod trained, predict returned f(2.0) = 7.00, and the Pod and endpoint were cleaned up.
Assumes runpod 2.0.0rc2, which carries the SDK fixes for these issues: - The build leaves torch out even when a dependency pulls it in, so golden path 08 lists peft and accelerate directly and the --no-deps workaround and its gotcha are gone. - GPU device names pin exact devices, so the set-endpoint-gpus advice is gone. - vol.path works on deployed workers. - Dev sessions clean up on kill or SIGINT, so the SIGINT wrapper is gone. - Only workers that call other resources get the API key. Install pins move to runpod>=2.0.0rc2. Concept rules are replaced to match.
justinwlin
force-pushed
the
docs/flash-to-runpod-sdk-2
branch
from
October 21, 2026 22:05
bcb59eb to
4cbbe30
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flash → SDK 2.0 fix stack
Independent PRs off
main; merge in this order. Status as of 2026-10-21.mainpeft/accelerategets bundledmainmainrp flash devleaks endpoints when stopped by a signalmainvol.pathsays "not mounted" on deployed workersmainrunpodwith 2–5dockercheck failing>=2.0.0rc2pin moves to it)Flash ships in
runpod2.0 as SDK apps (runpod.App,rp flash), andrunpod-flashis deprecated on October 29, 2026. Until this PR, the plugin taught onlyrunpod-flash.What changes
flashskill rewritten SDK-first. It checks which SDK a project is on, and new references cover migrating fromrunpod-flashand maintaining legacy projects.flash/examples/: five single-file apps, all passing live.flashconcept rules rewritten with public-doc and live evidence; the rules that cite them are updated.rp login/rp flash.No SDK workarounds left. With rc2:
peft/acceleratestays out of the bundle, so those are plaindependencies.vol.pathworks on deployed workers.killcleans up a background dev session.Remaining gotchas are behavior, not bugs: whole-number floats arrive as ints, and dev sessions run the runtime image's own
runpod.Live verification
2026-10-16,
runpod2.0.0rc1:rp flash dev --once2026-10-17, a local
runpodbuild with #623–#628 merged (1509 SDK tests pass):peft+accelerateas plain dependencies,vol.pathfor the adaptertorch 2.9.1+cu128, generated text. This run also wrapped the base-model load inVolumeCache(#90); here the base model loads from the mount as in the 2026-10-16 runAll test endpoints and volumes were deleted. Repo checks: ontology validate (0 errors), check:guides, links, branding, frontmatter, versions, marketplace, CLI absence claims, migrate checks.
How we tested and compared
Workarounds vs. no workarounds. Each SDK workaround the rc1 docs carried was first reproduced on rc1:
pip install --no-depsat runtime forpeft(rc1 crash-looped withlibcudart.so.13)/runpod-volumeinstead ofvol.path(rc1 raised "not mounted")kill(rc1 left the endpoint running)The same golden paths and examples were then re-run on a local build with #623–#628, with the workaround removed. The doc changed only where that run passed.
Docs vs. SDK source. A Codex review checked every SDK claim in the diff against the SDK at
v2.0.0rc1and the fix branches. Concept rules cite the public docs or the SDK source line.Review (Codex, two rounds)
.remote()has no timeout. On a queue, it stops waiting after 300 s (DEFAULT_TIMEOUT_SECONDS) while the job keeps runningapi.md,flash.yaml(flash.client-limits, with SDK evidence) and golden path 03-B now say 300 s on a queue and point to.spawn()+job.result(timeout=...)for longer calls. Task calls have no client timeout, so the claim is scoped to queues.flash.same-module-in-dev-and-deployscope the one-file rule to dev sessions, withdeploy.pyevidence.