Skip to content

test(cache): make stalled-remote-cache a proxy for the remote cache - #795

Merged
wan9chi merged 1 commit into
mainfrom
claude/stalled-remote-cache-proxy
Oct 4, 2026
Merged

wan9chi merged 1 commit into
mainfrom
claude/stalled-remote-cache-proxy

Conversation

@wan9chi

@wan9chi wan9chi commented Oct 4, 2026

Copy link
Copy Markdown
Member

Motivation

The e2e cases for a remote cache that never answers need some requests to stall while others behave normally. For example, an upload case needs its fetch to reach a backend and miss before the upload stalls. vtt stalled-remote-cache was its own endpoint and stalled every request, so it couldn't do that. #718 will replace remote-cache-server with the real cache service, which can't stall requests itself. So the stalling has to happen in front of whatever backend a test runs.

Changes

  • Proxy. vtt stalled-remote-cache [--stall ROUTE]... COMMAND [ARGS...] is now a proxy for the endpoint in VP_REMOTE_CACHE_URL, which must be http://<host>:<port>/<path>. It runs the command with VP_REMOTE_CACHE_URL set to the proxy.
    • A request to a stalled route below the endpoint, such as /store, is never answered, and it emits a stalled milestone.
    • Other requests are forwarded with connection: close, so each one comes in on its own connection and is checked on its own.
    • A forwarded request gets a 502 if the endpoint can't be reached.
  • Fetch cases. ctrl_c_during_fetch and fast_fail_during_fetch stall /fetch, with an unreachable endpoint behind the proxy. They still run on every platform without Node.js.
  • New ctrl_c_during_upload case. It runs remote-cache-server vtt stalled-remote-cache --stall /store vt run build. The fetch reaches the backend and misses, the upload stalls, and Ctrl-C cancels it.
  • remote-cache-server now ignores Ctrl-C and leaves it to its command, so it still prints its request log afterwards.

Notes for reviewers

🤖 Generated with Claude Code

vtt stalled-remote-cache used to stall every request to an endpoint of
its own. It is now a proxy for the endpoint in VP_REMOTE_CACHE_URL:
requests to the routes given with --stall are never answered, and the
rest are forwarded, so it can run in front of remote-cache-server or any
other backend. ctrl_c_during_fetch and fast_fail_during_fetch stall
/fetch with nothing behind the proxy. The new ctrl_c_during_upload case
stalls /store in front of the backend, which needs remote-cache-server
to leave Ctrl-C to its command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown

fspy benchmark

linux

dynamic/launch             change  +0.37%  [ -6.84% ..  +8.02%]  overhead  +269.24%
dynamic/access             change  +0.15%  [ -0.87% ..  +1.98%]  overhead   +13.49%
dynamic/access-relative    change  -0.11%  [ -1.30% ..  +0.87%]  overhead   +59.47%
dynamic/access-contended   change  -1.56%  [ -5.35% ..  +2.11%]  overhead   +14.39%
static/launch              change  +0.50%  [ -5.95% ..  +7.04%]  overhead  +727.56%
static/access              change  -0.15%  [ -1.44% ..  +0.97%]  overhead  +794.48%
static/access-relative     change  +0.15%  [ -1.05% ..  +1.97%]  overhead +1354.76%
static/access-contended    change  -0.31%  [ -2.00% ..  +0.48%]  overhead +3098.26%

macos

dynamic/launch             change  +0.71%  [ -2.16% ..  +4.68%]  overhead  +216.21%
dynamic/access             change  -0.70%  [-16.13% ..  +7.81%]  overhead    +3.01%
dynamic/access-relative    change  -1.44%  [-53.14% ..  +6.17%]  overhead  +271.54%
dynamic/access-contended   change  +1.28%  [ -4.54% ..  +6.50%]  overhead    +4.47%

windows

dynamic/launch             change  -0.22%  [ -4.99% ..  +6.48%]  overhead   +24.21%
dynamic/access             change  +1.58%  [ -4.30% .. +13.18%]  overhead    +2.34%
dynamic/access-relative    change  +0.00%  [ -9.69% .. +10.48%]  overhead    +1.66%
dynamic/access-contended   change  +1.09%  [ -7.30% ..  +7.59%]  overhead    +2.27%

@wan9chi
wan9chi merged commit 90d256f into main Oct 4, 2026
19 checks passed
@wan9chi
wan9chi deleted the claude/stalled-remote-cache-proxy branch October 4, 2026 06:32
wan9chi added a commit that referenced this pull request Oct 4, 2026
… HTTP/2 (#787)

## Motivation

In `read-write` mode, a task waited for its upload to the remote cache
before it counted as finished, so every task that depended on it waited
too. A slow remote cache slowed down the whole run, even though nothing
in the run needed the upload. Once uploads run side by side, HTTP/1.1
also opens a connection for each one; HTTP/2 lets them share one.

## Changes

- **Background uploads.** Once a task's result is cached locally, its
upload starts in the background and the task finishes right away. When
the graph is done, `vp run` prints `Waiting for N remote cache uploads
to finish (Ctrl-C to cancel)...` and waits for them before the summary.
An upload's error is set later, through the `Arc<OnceLock<UploadError>>`
in `CacheUpdateStatus::Updated`, and the summary reads it after the
wait.
- **Cancellation.** A new interrupt token, which only Ctrl-C cancels,
cancels the uploads. Their entries stay in the local cache, and the
summary says they weren't uploaded because they were interrupted.
Fast-fail still stops lookups but no longer stops uploads, so tasks that
succeeded before another one failed are still uploaded.
`docs/cancellation.md` covers both.
- **HTTP/2.** reqwest's `http2` feature is on. HTTPS endpoints use
HTTP/2 if the server accepts it in the TLS handshake and HTTP/1.1
otherwise. `http://` endpoints stay on HTTP/1.1.
- **Tests.** Whether an upload is still running when the graph is done
depends on how fast the remote cache responds. So e2e steps that upload
set `VP_RUN_INTERNAL_HIDE_PENDING_UPLOADS`, which keeps the waiting line
out of their output, including the first step of `restore_failure` from
#770. The cases that show the waiting line or cancel uploads put `vtt
stalled-remote-cache --stall /store` from #795 in front of the backend.
Their fetches reach the backend and miss, and their uploads never
finish. Like the other backend cases, they need Node.js and are skipped
on Windows. New e2e cases: `pending_uploads`, `hide_pending_uploads`,
and `fast_fail_during_upload`. `ctrl_c_during_upload` from #795 now
shows the waiting line. A unit test checks that the client offers `h2`
over TLS.

## Notes for reviewers

- **Uploads aren't capped.** Each upload in progress holds its encoded
entry in memory and its output archive open, plus its own connection on
HTTP/1.1, and none of that counts against `--concurrency-limit`. A large
graph with a slow endpoint can pile them up. A cap is left for a
follow-up.
- **No HTTP/3.** reqwest's `http3` feature needs `--cfg
reqwest_unstable` in every build, vite-plus included, adds aws-lc-rs
next to ring, bypasses proxies, and is used only when the client forces
HTTP/3 for every request.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant