test: hermetic e2e suite for Altimate Base (registration, catalog, inference, rate-limit/budget, error surfacing) - #1248
Conversation
Foundation for a 6-suite parallel Altimate Base e2e test partition (see the design doc). Adds `test/altimate/_fixtures/fake-gateway.ts` (a `FakeGateway` that intercepts `fetch` via `spyOn(globalThis, "fetch")` — the repo's existing proven pattern, not a real HTTP server — implementing `/register` and `/v1/chat/completions` with controllable knobs for every failure mode the suites need: per-minute token rate-limit, both `budget_exceeded` variants, request-too-large, 401, 5xx, timeout, malformed JSON, and success) and `test/altimate/_fixtures/altimate-base-harness.ts` (isolated XDG/home bootstrap + gateway-env reset helpers, extracted from `altimate-base.test.ts`'s existing pattern so every suite shares one implementation). Adds `altimate-base-harness-smoke.test.ts` proving the harness works in both directions: a register -> `authorizedFetch` happy-path round trip, and one scripted failure knob (per-minute token rate-limit -> `describeRateLimit`'s non-retryable message). Does not add any of the 6 planned suite files themselves — those are a separate, parallel follow-up. Copies the design doc (`docs/internal/2026-09-04-altimate-base-e2e-harness-plan.md`) into the branch so it travels with the PR. Stacked on `codex/altimate-base-release-final` (#1199) since the harness targets that branch's 131072/65536 limits and Altimate Base code.
…rming Brings together 5 independently-written hermetic e2e suites for Altimate Base onto the shared harness branch (53 tests): - `altimate-base-registration-gaps.test.ts` (11) — HTTP/network/malformed register failure mapping, payload shape, retry idempotency - `altimate-base-catalog.test.ts` (9) — model catalog / provider isolation - `altimate-base-inference-e2e.test.ts` (5) — register -> list -> fetch round trip, placeholder-vs-real-key isolation - `altimate-base-rate-limit-messages.test.ts` (21) — throttle/budget/ request-too-large message mapping - `altimate-base-error-surfacing.test.ts` (7) — 5xx/timeout/abort/ malformed-body/401 pass-through at the inference layer All 5 (plus the two pre-existing files, `altimate-base.test.ts` and `altimate-base-harness-smoke.test.ts`) independently called `FreeTierCapability.issueArmer()` at module scope. That capability is process-global and throws on a second call, so running the directory in one `bun test` invocation — as CI does — threw "Altimate Base consent armer already issued for this process" once a second armer-calling file loaded into the same worker process (reproducible with just the two pre-existing files, before any of these suites existed). Fix: centralize arming in the shared harness (`_fixtures/altimate-base-harness.ts`) behind a new `consented()` helper that lazily calls `issueArmer()` exactly once per process and caches the returned armer in a module-level singleton. Because bun caches modules per process, every suite file that imports `consented()` shares that one cached armer regardless of load order or file count. This adds no way to reset, re-claim, or otherwise weaken the one-shot guarantee `issueArmer()` already enforces — it is a cache in front of the single legitimate call, not a new capability. All 7 armer-calling files now import and use the shared helper instead of claiming their own. Verified with `bun test --timeout 90000 test/altimate/` (the directory CI covers, at CI's timeout) from `packages/opencode`: 5103 pass, 0 fail, zero armer-collision errors, in one process invocation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_e9e0cb77-cc9e-4751-b90e-c28b01e5b412) |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Team Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
full receipts (3 sessions)
builder ·
|
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
2 similar comments
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d2973f4ec9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // kicking off the call could fire before the fake gateway's `timeout` branch has attached its | ||
| // `abort` listener. `AbortSignal.timeout` schedules the abort on a real timer instead, so it | ||
| // always fires after the listener is attached. | ||
| const promise = FreeTier.authorizedFetch(url, { ...init, signal: AbortSignal.timeout(50) }) |
There was a problem hiding this comment.
On a slow CI filesystem, credentialsForLoad() can take longer than 50 ms, so this signal may abort before FakeGateway.handleChat() installs its abort listener. The fake does not check signal.aborted when entering timeout mode, leaving its promise pending until Bun's 30-second test timeout. Publish gateway readiness and abort afterward, or make the fake immediately reject an already-aborted signal.
AGENTS.md reference: packages/opencode/test/AGENTS.md:L165-L169
Useful? React with 👍 / 👎.
| function armer(): (token: string) => void { | ||
| if (!cachedArmer) cachedArmer = FreeTierCapability.issueArmer() | ||
| return cachedArmer |
There was a problem hiding this comment.
Because the shared armer is now claimed lazily, running only the existing unforgeable consent: no in-process caller can mint an independent authority test means no earlier call to consented() has occurred. Its direct FreeTierCapability.issueArmer() call therefore succeeds even though the test expects it to throw, so common bun test -t ... workflows fail. Claim the armer during module setup or explicitly initialize it in that test before checking the second-claim behavior.
Useful? React with 👍 / 👎.
| const response = await FreeTier.authorizedFetch(`${GATEWAY_URL}/v1/chat/completions`, chatRequestInit()) | ||
|
|
||
| expect(response.status).toBe(200) | ||
| const body = (await response.json()) as { choices: [{ message: { content: string } }] } |
There was a problem hiding this comment.
This purported inference E2E path calls FreeTier.authorizedFetch directly and manually parses the response, so it never constructs the @ai-sdk/openai-compatible model or exercises Provider.getModel/Provider.getLanguage, request serialization, model selection, and SDK response decoding. A regression that leaves the fetch function present in provider options but makes the configured model unusable would therefore pass the entire new suite; drive a generation through the provider model, as other provider E2E tests do, instead of invoking the transport seam directly.
Useful? React with 👍 / 👎.
| const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url | ||
| if (url.endsWith("/register")) return this.handleRegister(url, init) | ||
| if (url.includes("/v1/chat/completions")) return this.handleChat(url, init) | ||
| throw new Error(`FakeGateway: unhandled URL ${url}`) |
There was a problem hiding this comment.
The fake dispatches only by URL suffix/substring and never checks init.method, so it accepts requests that the real gateway rejects—for example, a registration accidentally changed from POST to GET, or a chat request sent to /v1/chat/completions-invalid. Because replacing global fetch also bypasses native rejection of a GET request with a body, the registration contract tests can remain green while the shipped client fails before reaching the gateway. Match the exact pathname and require POST for both routes.
Useful? React with 👍 / 👎.
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
| import fs from "node:fs" | ||
| import os from "node:os" | ||
| import path from "node:path" | ||
| import { consented } from "./_fixtures/altimate-base-harness" |
There was a problem hiding this comment.
SUGGESTION: Redundant environment-isolation code — consolidate onto the shared helper.
This file still hand-rolls the XDG/home isolation (isolatedEnvironment, originalEnvironment, temporaryHome, and the afterAll cleanup at lines 8-21 and 68-75), duplicating what isolateAltimateBaseHome() in _fixtures/altimate-base-harness.ts now provides. The harness plan (docs/internal/2026-09-04-altimate-base-e2e-harness-plan.md) explicitly intended altimate-base.test.ts to import the shared bootstrap so there is "exactly one isolated-environment implementation", but only consented() was migrated. Replace the inline block with isolateAltimateBaseHome("altimate-base").
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
| claimable exactly once **per process**, not per file. `bun test` runs each test file in its own | ||
| worker process by default (confirmed by the existing suite's comment at `altimate-base.test.ts:76-82` | ||
| treating this as safe), so each suite file gets its own fresh module instances and can safely call | ||
| `FreeTierCapability.issueArmer()` at module scope, exactly like the existing file does. **Do not** |
There was a problem hiding this comment.
SUGGESTION: This guidance contradicts the shipped fix and would reintroduce the crash.
This section instructs each suite to call FreeTierCapability.issueArmer() at module scope on the premise that "bun test runs each test file in its own worker process." The actual fix in this PR (consented() in _fixtures/altimate-base-harness.ts) exists precisely because multiple suite files load into one worker, where a second module-scope issueArmer() throws. A future implementer following this section (or the example test later in this file that calls issueArmer() directly) would reintroduce the "consent armer already issued" crash. Update this section to direct suites to use consented() instead.
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
Code Review SummaryStatus: 2 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)SUGGESTION
Files Reviewed (10 files)
Fix these issues in Kilo Cloud Reviewed by deepseek-v4-pro · Input: 96.3K · Output: 32.8K · Cached: 1.3M Review guidance: REVIEW.md from base branch |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
8 issues found across 10 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/opencode/test/altimate/_fixtures/altimate-base-harness.ts">
<violation number="1" location="packages/opencode/test/altimate/_fixtures/altimate-base-harness.ts:59">
P3: resetGatewayEnv() permanently mutates process env: it deletes ALTIMATE_FREE_GATEWAY_URL and sets ALTIMATE_BASE_GATEWAY_URL to the fake gateway URL, but no teardown restores the prior values. isolateAltimateBaseHome()'s afterAll only restores the ISOLATED_ENV keys, so in a shared `bun test` worker the fake gateway URL stays set and the legacy var stays deleted after the suite. Follow the file's own restoration pattern: capture the original values once (e.g., alongside the ISOLATED_ENV snapshot in isolateAltimateBaseHome) and restore them in the afterAll.</violation>
<violation number="2" location="packages/opencode/test/altimate/_fixtures/altimate-base-harness.ts:79">
P2: When the unforgeable-consent test runs alone with `bun test -t`, `cachedArmer` is still unset, so its direct `issueArmer()` call succeeds instead of throwing. Claim the shared armer during module setup so filtered tests still exercise a second claim.</violation>
</file>
<file name="docs/internal/2026-09-04-altimate-base-e2e-harness-plan.md">
<violation number="1" location="docs/internal/2026-09-04-altimate-base-e2e-harness-plan.md:3">
P3: The doc's status header says "Not yet implemented" and describes itself as the contract for implementer agents, but every suite and fixture it specifies is implemented and shipped in this same PR. Its Deliverable 2 example and "Cross-file consent isolation" section also instruct each file to claim `issueArmer()` once at module scope, whereas the shipped harness uses a shared `consented()` singleton that claims the process's single armer — the exact collision this PR was built to fix. Update the status to reflect that the design is implemented and that the consent helper is a per-process `consented()` shared across files, not a per-file `issueArmer()`, so a future reader doesn't follow the stale design.</violation>
<violation number="2" location="docs/internal/2026-09-04-altimate-base-e2e-harness-plan.md:360">
P2: The fake gateway accepts invalid methods and paths, allowing contract tests to pass for requests the real gateway rejects. Dispatch only when the exact pathname is paired with `POST` for both routes.</violation>
</file>
<file name="packages/opencode/test/altimate/_fixtures/fake-gateway.ts">
<violation number="1" location="packages/opencode/test/altimate/_fixtures/fake-gateway.ts:151">
P3: The `timeout` chat mode never settles when the request has no AbortSignal, and also hangs if the signal is already aborted before the listener attaches. Check the signal and reject immediately in those cases so a mis-scripted test degrades to a clear rejection instead of a hang until Bun's global timeout.</violation>
</file>
<file name="packages/opencode/test/altimate/altimate-base-registration-gaps.test.ts">
<violation number="1" location="packages/opencode/test/altimate/altimate-base-registration-gaps.test.ts:143">
P3: The primary assertion in the cli_version test is a tautology: `sentVersion` is computed by the client as exactly `sanitizeCliVersion(Installation.VERSION)`, so comparing it to the same re-called function can never fail independently and gives false confidence that the sanitizer contract is being exercised. The only meaningful check is the following regex. Make the assertion self-contained by hardcoding an expected value, or rely on the regex alone and drop the self-comparison.</violation>
</file>
<file name="packages/opencode/test/altimate/altimate-base.test.ts">
<violation number="1" location="packages/opencode/test/altimate/altimate-base.test.ts:6">
P3: Use `isolateAltimateBaseHome("altimate-base")` here and remove the inline environment setup so this file has the same single isolation implementation as the new suites.</violation>
</file>
<file name="packages/opencode/test/altimate/altimate-base-inference-e2e.test.ts">
<violation number="1" location="packages/opencode/test/altimate/altimate-base-inference-e2e.test.ts:84">
P2: This test bypasses the configured `@ai-sdk/openai-compatible` model, so it cannot catch broken model construction, provider model selection, request serialization, or SDK response decoding. Drive a generation through the provider model instead.</violation>
</file>
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
| // otherwise bypass that one-shot guarantee. It is purely a shared cache in front of the single | ||
| // legitimate call, so the underlying security property (only one in-process caller can ever obtain | ||
| // the ability to arm the production consent authority) is unchanged. | ||
| let cachedArmer: ((token: string) => void) | undefined |
There was a problem hiding this comment.
P2: When the unforgeable-consent test runs alone with bun test -t, cachedArmer is still unset, so its direct issueArmer() call succeeds instead of throwing. Claim the shared armer during module setup so filtered tests still exercise a second claim.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/test/altimate/_fixtures/altimate-base-harness.ts, line 79:
<comment>When the unforgeable-consent test runs alone with `bun test -t`, `cachedArmer` is still unset, so its direct `issueArmer()` call succeeds instead of throwing. Claim the shared armer during module setup so filtered tests still exercise a second claim.</comment>
<file context>
@@ -0,0 +1,95 @@
+// otherwise bypass that one-shot guarantee. It is purely a shared cache in front of the single
+// legitimate call, so the underlying security property (only one in-process caller can ever obtain
+// the ability to arm the production consent authority) is unchanged.
+let cachedArmer: ((token: string) => void) | undefined
+
+function armer(): (token: string) => void {
</file context>
| if (url.endsWith("/register")) return this.handleRegister(url, init) | ||
| if (url.includes("/v1/chat/completions")) return this.handleChat(url, init) |
There was a problem hiding this comment.
P2: The fake gateway accepts invalid methods and paths, allowing contract tests to pass for requests the real gateway rejects. Dispatch only when the exact pathname is paired with POST for both routes.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At docs/internal/2026-09-04-altimate-base-e2e-harness-plan.md, line 360:
<comment>The fake gateway accepts invalid methods and paths, allowing contract tests to pass for requests the real gateway rejects. Dispatch only when the exact pathname is paired with `POST` for both routes.</comment>
<file context>
@@ -0,0 +1,672 @@
+ this.spy = spyOn(globalThis, "fetch").mockImplementation(
+ (async (input: RequestInfo | URL, init?: RequestInit) => {
+ const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url
+ if (url.endsWith("/register")) return this.handleRegister(url, init)
+ if (url.includes("/v1/chat/completions")) return this.handleChat(url, init)
+ throw new Error(`FakeGateway: unhandled URL ${url}`)
</file context>
| if (url.endsWith("/register")) return this.handleRegister(url, init) | |
| if (url.includes("/v1/chat/completions")) return this.handleChat(url, init) | |
| const request = new URL(url) | |
| if (request.pathname === "/register" && init?.method === "POST") return this.handleRegister(url, init) | |
| if (request.pathname === "/v1/chat/completions" && init?.method === "POST") return this.handleChat(url, init) |
| await registerWithGateway() | ||
|
|
||
| gateway.chatNext({ kind: "ok", content: "the answer is 42" }) | ||
| const response = await FreeTier.authorizedFetch(`${GATEWAY_URL}/v1/chat/completions`, chatRequestInit()) |
There was a problem hiding this comment.
P2: This test bypasses the configured @ai-sdk/openai-compatible model, so it cannot catch broken model construction, provider model selection, request serialization, or SDK response decoding. Drive a generation through the provider model instead.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/test/altimate/altimate-base-inference-e2e.test.ts, line 84:
<comment>This test bypasses the configured `@ai-sdk/openai-compatible` model, so it cannot catch broken model construction, provider model selection, request serialization, or SDK response decoding. Drive a generation through the provider model instead.</comment>
<file context>
@@ -0,0 +1,185 @@
+ await registerWithGateway()
+
+ gateway.chatNext({ kind: "ok", content: "the answer is 42" })
+ const response = await FreeTier.authorizedFetch(`${GATEWAY_URL}/v1/chat/completions`, chatRequestInit())
+
+ expect(response.status).toBe(200)
</file context>
| export function resetGatewayEnv(gatewayUrl: string): void { | ||
| delete process.env.ALTIMATE_BASE_GATEWAY_URL | ||
| delete process.env.ALTIMATE_FREE_GATEWAY_URL | ||
| process.env.ALTIMATE_BASE_GATEWAY_URL = gatewayUrl |
There was a problem hiding this comment.
P3: resetGatewayEnv() permanently mutates process env: it deletes ALTIMATE_FREE_GATEWAY_URL and sets ALTIMATE_BASE_GATEWAY_URL to the fake gateway URL, but no teardown restores the prior values. isolateAltimateBaseHome()'s afterAll only restores the ISOLATED_ENV keys, so in a shared bun test worker the fake gateway URL stays set and the legacy var stays deleted after the suite. Follow the file's own restoration pattern: capture the original values once (e.g., alongside the ISOLATED_ENV snapshot in isolateAltimateBaseHome) and restore them in the afterAll.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/test/altimate/_fixtures/altimate-base-harness.ts, line 59:
<comment>resetGatewayEnv() permanently mutates process env: it deletes ALTIMATE_FREE_GATEWAY_URL and sets ALTIMATE_BASE_GATEWAY_URL to the fake gateway URL, but no teardown restores the prior values. isolateAltimateBaseHome()'s afterAll only restores the ISOLATED_ENV keys, so in a shared `bun test` worker the fake gateway URL stays set and the legacy var stays deleted after the suite. Follow the file's own restoration pattern: capture the original values once (e.g., alongside the ISOLATED_ENV snapshot in isolateAltimateBaseHome) and restore them in the afterAll.</comment>
<file context>
@@ -0,0 +1,95 @@
+export function resetGatewayEnv(gatewayUrl: string): void {
+ delete process.env.ALTIMATE_BASE_GATEWAY_URL
+ delete process.env.ALTIMATE_FREE_GATEWAY_URL
+ process.env.ALTIMATE_BASE_GATEWAY_URL = gatewayUrl
+}
+
</file context>
| @@ -0,0 +1,672 @@ | |||
| # Altimate Base — E2E Test Suite: Spec, Harness Design, Parallel Partition | |||
|
|
|||
| Status: Phase 1 (research + design) complete. Not yet implemented. | |||
There was a problem hiding this comment.
P3: The doc's status header says "Not yet implemented" and describes itself as the contract for implementer agents, but every suite and fixture it specifies is implemented and shipped in this same PR. Its Deliverable 2 example and "Cross-file consent isolation" section also instruct each file to claim issueArmer() once at module scope, whereas the shipped harness uses a shared consented() singleton that claims the process's single armer — the exact collision this PR was built to fix. Update the status to reflect that the design is implemented and that the consent helper is a per-process consented() shared across files, not a per-file issueArmer(), so a future reader doesn't follow the stale design.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At docs/internal/2026-09-04-altimate-base-e2e-harness-plan.md, line 3:
<comment>The doc's status header says "Not yet implemented" and describes itself as the contract for implementer agents, but every suite and fixture it specifies is implemented and shipped in this same PR. Its Deliverable 2 example and "Cross-file consent isolation" section also instruct each file to claim `issueArmer()` once at module scope, whereas the shipped harness uses a shared `consented()` singleton that claims the process's single armer — the exact collision this PR was built to fix. Update the status to reflect that the design is implemented and that the consent helper is a per-process `consented()` shared across files, not a per-file `issueArmer()`, so a future reader doesn't follow the stale design.</comment>
<file context>
@@ -0,0 +1,672 @@
+# Altimate Base — E2E Test Suite: Spec, Harness Design, Parallel Partition
+
+Status: Phase 1 (research + design) complete. Not yet implemented.
+Scope: PR #1199, branch `codex/altimate-base-release-final`.
+Author: research/design pass, 2026-09-04. No test code was written by this pass — this
</file context>
| return new Response("", { status: 401 }) | ||
| case "server-error": | ||
| return new Response("upstream error", { status: mode.status ?? 500 }) | ||
| case "timeout": |
There was a problem hiding this comment.
P3: The timeout chat mode never settles when the request has no AbortSignal, and also hangs if the signal is already aborted before the listener attaches. Check the signal and reject immediately in those cases so a mis-scripted test degrades to a clear rejection instead of a hang until Bun's global timeout.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/test/altimate/_fixtures/fake-gateway.ts, line 151:
<comment>The `timeout` chat mode never settles when the request has no AbortSignal, and also hangs if the signal is already aborted before the listener attaches. Check the signal and reject immediately in those cases so a mis-scripted test degrades to a clear rejection instead of a hang until Bun's global timeout.</comment>
<file context>
@@ -0,0 +1,174 @@
+ return new Response("", { status: 401 })
+ case "server-error":
+ return new Response("upstream error", { status: mode.status ?? 500 })
+ case "timeout":
+ return new Promise<Response>((_resolve, reject) => {
+ init?.signal?.addEventListener("abort", () => reject(init.signal?.reason), { once: true })
</file context>
| case "timeout": | |
| case "timeout": { | |
| const signal = init?.signal | |
| if (signal?.aborted) return Promise.reject(signal.reason) | |
| return new Promise<Response>((_resolve, reject) => { | |
| signal?.addEventListener("abort", () => reject(signal.reason), { once: true }) | |
| }) | |
| } |
|
|
||
| expect(gateway.registerCalls).toHaveLength(1) | ||
| const sentVersion = gateway.registerCalls[0]!.cliVersion | ||
| expect(sentVersion).toBe(FreeTier.sanitizeCliVersion(Installation.VERSION)) |
There was a problem hiding this comment.
P3: The primary assertion in the cli_version test is a tautology: sentVersion is computed by the client as exactly sanitizeCliVersion(Installation.VERSION), so comparing it to the same re-called function can never fail independently and gives false confidence that the sanitizer contract is being exercised. The only meaningful check is the following regex. Make the assertion self-contained by hardcoding an expected value, or rely on the regex alone and drop the self-comparison.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/test/altimate/altimate-base-registration-gaps.test.ts, line 143:
<comment>The primary assertion in the cli_version test is a tautology: `sentVersion` is computed by the client as exactly `sanitizeCliVersion(Installation.VERSION)`, so comparing it to the same re-called function can never fail independently and gives false confidence that the sanitizer contract is being exercised. The only meaningful check is the following regex. Make the assertion self-contained by hardcoding an expected value, or rely on the regex alone and drop the self-comparison.</comment>
<file context>
@@ -0,0 +1,176 @@
+
+ expect(gateway.registerCalls).toHaveLength(1)
+ const sentVersion = gateway.registerCalls[0]!.cliVersion
+ expect(sentVersion).toBe(FreeTier.sanitizeCliVersion(Installation.VERSION))
+ // sanitizeCliVersion's contract: only these characters survive, capped at 32 chars, never empty.
+ expect(sentVersion).toMatch(/^[A-Za-z0-9._+-]{1,32}$/)
</file context>
| import fs from "node:fs" | ||
| import os from "node:os" | ||
| import path from "node:path" | ||
| import { consented } from "./_fixtures/altimate-base-harness" |
There was a problem hiding this comment.
P3: Use isolateAltimateBaseHome("altimate-base") here and remove the inline environment setup so this file has the same single isolation implementation as the new suites.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/test/altimate/altimate-base.test.ts, line 6:
<comment>Use `isolateAltimateBaseHome("altimate-base")` here and remove the inline environment setup so this file has the same single isolation implementation as the new suites.</comment>
<file context>
@@ -3,6 +3,7 @@ import { createHash, randomBytes } from "node:crypto"
import fs from "node:fs"
import os from "node:os"
import path from "node:path"
+import { consented } from "./_fixtures/altimate-base-harness"
const isolatedEnvironment = [
</file context>
1 participant
Issue for this PR
Closes #1247
Type of change
What does this PR do?
Adds 53 hermetic e2e tests across 5 suites for Altimate Base's free-tier client (
src/altimate/free/*), consolidated onto the shared_fixtures/altimate-base-harness.ts+_fixtures/fake-gateway.tsharness already on this branch:altimate-base-registration-gaps.test.ts(11) — HTTP 4xx/5xx/network/malformed-JSON register failure mapping ontoRegistrationError, the exact request payload (hashed install secret,cli_version), and retry idempotency.altimate-base-catalog.test.ts(9) — model catalog / provider isolation (Provider.list()/Provider.defaultModel()/Provider.sort()) against a credential minted through the real consent + register path.altimate-base-inference-e2e.test.ts(5) — full register → provider-list →authorizedFetchround trip, plus the placeholder-vs-real-key and credential-storage isolation properties.altimate-base-rate-limit-messages.test.ts(21) — everydescribeRateLimit/describeRequestTooLargebranch: per-minute token throttle, burst throttle, wallet/global/unknown budget, request-too-large byte math, and the malformed/unrecognized fallback paths.altimate-base-error-surfacing.test.ts(7) — non-rate-limit inference-time failures reachauthorizedFetch's caller cleanly: 5xx pass-through, timeout/abort propagation, raw connection failure, malformed JSON body, and chat-time 401.All suites are fully hermetic —
fetchis injected via the sharedFakeGatewayfixture (spyOn(globalThis, "fetch")), so there is no live gateway, no credentials, and no network access anywhere in this PR. Each suite runs in its own isolated XDG/home tree (isolateAltimateBaseHome) so credential stores, config, and cache never collide across files or touch a real user directory. This is picked up by CI automatically via the existingtest/altimate/**path filters — no workflow change needed.The centralized-arming fix (the load-bearing change in this PR):
FreeTierCapability.issueArmer()is a process-global capability that throws on a second call in the same process (by design — it's the security property that makes Altimate Base's consent gate unforgeable, seesrc/altimate/free/capability.ts). Every one of these 5 new suites, plus the two pre-existing files (altimate-base.test.ts,altimate-base-harness-smoke.test.ts), independently calledissueArmer()at module scope.bun test test/altimate/loads multiple test files into one worker process, so this throws"Altimate Base consent armer already issued for this process"as soon as a second armer-calling file loads — reproducible with just the two pre-existing files, before any of these suites existed.Fixed by adding a
consented()helper to the shared harness (_fixtures/altimate-base-harness.ts) that lazily claimsissueArmer()exactly once per process and caches the returned armer in a module-level singleton. Because bun caches modules per process, every suite file that importsconsented()— regardless of load order or how many files load it — shares that one cached armer. All 7 armer-calling files (5 new + 2 existing) now go through this shared helper instead of claiming their own armer. This adds no way to reset, re-claim, or otherwise weaken the one-shot guaranteeissueArmer()already enforces; it is purely a cache in front of the single legitimate call, so the underlying security property (only one in-process caller can ever obtain the ability to arm the production consent authority) is unchanged — the existing "unforgeable consent" test inaltimate-base.test.ts(which asserts a secondissueArmer()/issueRedeemer()call throws) still passes unmodified.Live-prod smoke testing against a real gateway is intentionally out of scope here and tracked separately / non-blocking for this PR — everything in this PR is fetch-injected and hermetic.
This PR stacks on #1199 (Altimate Base hosted model release) — the base is
codex/altimate-base-release-finalso the diff here shows only the harness + tests, not #1199's feature changes. It should be retargeted tomainonce #1199 merges.How did you verify your code works?
Ran the full
test/altimate/directory in onebun testinvocation, matching CI's own invocation and timeout (bun test --timeout 90000, frompackages/opencode, per.github/workflows/ci.yml'stypescriptjob):Zero "consent armer already issued" errors — this is the actual acceptance bar, not file-by-file green (each file was also independently confirmed green: 11 + 9 + 5 + 21 + 7 = 53 new tests, all passing).
Also verified:
bun run typecheck(bun turbo typecheck) — clean, 13/13 tasks successful.bun run script/upstream/analyze.ts --markers --base origin/main --strict— clean, no unmarked upstream-shared changes.bunx prettier --checkon all changed files — clean.Screenshots / recordings
N/A — test-only change, no UI.
Checklist
Note
Low Risk
Test-only changes with no runtime behavior modifications; the shared
consented()helper preserves the one-shotissueArmer()security model while fixing multi-file test loading.Overview
Adds a hermetic Altimate Base e2e test layer built on shared
_fixtures/altimate-base-harness.ts(isolated XDG home, gateway env reset) and_fixtures/fake-gateway.ts(spyOn(globalThis, "fetch")for/registerand chat completions with scripted error modes). Five new suites cover registration failure gaps, provider catalog/defaultModel()behavior, register→Provider.list()→authorizedFetchinference, fulldescribeRateLimit/describeRequestTooLargebranches, and inference-time 5xx/timeout/network/malformed-JSON/401 surfacing; a small harness smoke file exercises the fixtures.Load-bearing harness fix: all suites (and the existing
altimate-base.test.ts) now mint consent via sharedconsented(), which lazily claimsFreeTierCapability.issueArmer()once per process sobun test test/altimate/does not crash when multiple files load in one worker.Also adds
docs/internal/2026-09-04-altimate-base-e2e-harness-plan.mdas the design contract. The planned context-clamp suite is not in this diff (still flagged in the doc). No production source changes; tests only, no CI workflow edits.Reviewed by Cursor Bugbot for commit d2973f4. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by cubic
Adds 53 hermetic e2e tests across 5 suites for Altimate Base's free-tier client (closes #1247), and centralizes the process-global
issueArmer()capability behind a sharedconsented()helper so all test files run together in one worker process without the "consent armer already issued" crash.Coverage
fetchvia the sharedFakeGatewayfixture with isolated XDG/home trees — no network, credentials, or live gateway.bun testinvocation at CI's timeout (5103 pass, 0 fail), and the diff is test-only since it stacks on feat: release Altimate Base hosted model #1199 — retarget tomainonce that merges.Consent armer
issueArmer()throws on a second in-process call by design; 7 files each claiming it at module scope crashed under bun's single-worker test loading.consented()lazily claims the armer once per process and caches it, so all 7 files share it — the unforgeable-consent guarantee is unchanged and still covered by an existing test.Written for commit d2973f4. Summary will update on new commits.