Skip to content

fix(apps): give the account api key only to workers that call resources - #628

Closed
justinwlin wants to merge 4 commits into
mainfrom
fix/scope-worker-api-key
Closed

justinwlin wants to merge 4 commits into
mainfrom
fix/scope-worker-api-key

Conversation

@justinwlin

@justinwlin justinwlin commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Flash → SDK 2.0 fix stack

PRs 1–9 are independent PRs off main, and 10 is stacked on 9. Merge in this order. Hover a link for its status.

# PR Fixes Needs
1 #627 ✅ merged: CI e2e fixed on main —
2 #623 GPU crash loop: torch pulled in by peft/accelerate gets bundled nothing
3 #624 Workers land on GPUs you didn't request (whole pools) nothing
4 #625 rp flash dev leaks endpoints when stopped by a signal nothing
5 #626 vol.path says "not mounted" on deployed workers nothing
6 👉 #628 (this PR) Account API key copied into every endpoint nothing
7 — Release runpod 2.0.0rc2 with 1–6 1–6 merged
8 runpod-workers/sdk-runtime#1 Rebuild runtime images (dev sessions run an old SDK) independent
9 runpod/runpod-plugins-official#89 Plugin docs for the fixed SDK (flash lane → SDK 2.0) 2.0.0rc2 on PyPI
10 runpod/runpod-plugins-official#90 Plugin VolumeCache guidance: global volumes, measured numbers, golden paths 08/09 (stacked on #89) runpod/runpod-plugins-official#89

rp flash deploy and rp flash dev copy the account API key into every endpoint's template env. Workers that never call another resource still hold full account credentials, and so does anyone who can read the template. runpod-flash injected the key only for endpoints that make remote calls.

Change

New runpod/apps/remote_calls.py. For each resource it parses the module source, walks the resource body (skipping decorators, so @runpod.post doesn't count) and the module-level helpers it reaches. It gives the key when that code:

  • references another resource, or calls .remote / .spawn
  • uses anything imported from runpod (for example runpod.Queue(...))
  • uses a project-local import it can't follow, at module level or inside the body
  • is reached from a queue @handle.init hook
  • references a handle from any app in the module

Unreadable source and nested definitions also get the key. Every uncertain case gives the key, so a worker that needs it never goes without. Deploy and dev payloads both use the analysis. Redeploying rewrites the template env, which drops a key left by an earlier deploy.

Testing

Endpoint What it does main runpod-flash 1.21 This PR (live)
leaf no remote calls key in template no key (injects only for remote-call endpoints) has_key: False
caller leaf.remote() key key has_key: True; nested call returns
via_helper from helpers import call_leaf inside the body; the helper uses runpod.Queue(...) key not tested has_key: True; nested call returns {'has_key': False, 'x': 10}
warm @warm.init hook calls leaf.remote() key n/a has_key: True; the init call returned {'has_key': False, 'x': 2}
How we tested and compared
  • Live, three builds of the same app. Each endpoint reports bool(os.environ.get("RUNPOD_API_KEY")) and the nested result.
    • This PR, on top of current main with the other fixes: the results in the table.
    • The first version of this PR (3817ba1), deployed as a separate app so every worker was fresh: leaf had no key (correct), via_helper failed with no api key configured. run rp login or set RUNPOD_API_KEY., and warm never served a job (IN_QUEUE for 240 s, one worker UNHEALTHY). These are the cases the review found.
    • The earlier leaf/caller deploy: {'has_key': True, 'nested': {'has_key': False, 'x': 42}}.
  • All test apps were undeployed and deleted.
  • Unit tests classify a direct call, a call through a helper, a runpod.Queue client, an API class with only decorators, a local import, a nested definition, an import inside the body (plus stdlib and client-absent packages, which don't count), a package import from inside the package, a queue init hook, and another app's handle. Deploy and dev payload env are also checked. pytest tests: 1488 passed, coverage 91.54%.
Review (Codex, two rounds)
Finding Resolution
An import inside the resource body wasn't seen Fixed in 3e3a16c. Body imports of runpod or project modules give the key.
from mypkg.helpers import ... inside mypkg/ resolved against the wrong directory Fixed in 3e3a16c. Locality is checked against the module directory, the package root, and the working directory, then against installed locations.
Queue @handle.init hooks weren't analyzed Fixed in 3e3a16c. The hook is an analysis root.
A handle from another app in the same module wasn't recognized Fixed in 3e3a16c. Any handle in the module's globals counts.
The four new cases are tested on the classifier, not the payload Open. Payload building calls the same classifier, and the payload tests cover the existing cases.

Deploy and dev copied the client's account API key into every
endpoint's template env, so any endpoint's workers (and anyone who
can read its template) held full account credentials, including
endpoints that never call another resource. runpod-flash injected the
key only for endpoints whose functions make remote calls.

Analyze each resource's module source: the resource body (not its
decorators) and the module-level helpers it reaches. The key is added
when that code references another resource, calls .remote or .spawn,
uses anything imported from runpod, or uses a project-local import the
analysis cannot follow. Unreadable source and nested definitions also
get the key, so a worker that needs it never goes without.
…andles

the analysis missed four ways a worker reaches another resource: an import inside the resource body, a package import resolved from inside the package, a queue init hook, and a handle from another app in the module. each now gives the key.
@justinwlin
justinwlin force-pushed the fix/scope-worker-api-key branch from 8953786 to 3e3a16c Compare October 20, 2026 15:48
@justinwlin
justinwlin marked this pull request as ready for review October 20, 2026 15:48
@KAJdev

KAJdev commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

this is a duplicate of #641

@justinwlin justinwlin closed this Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants