feat: add shared alpha extension authentication and custom CA transport

[pimlock]

Problem Statement

OpenShell needs an alpha authentication mechanism for extension services so middleware and gateway-interceptor integrations can begin implementing against a stable contract without waiting for production-grade mTLS, multi-key rotation, or per-runtime secret delivery. The mechanism should reuse the gateway's existing Ed25519 token minting and supervisor token-refresh APIs wherever possible, and remote middleware must support TLS verification with an operator-provided CA.

This spike narrows #2430 to an integration-enabling slice. The implementation must use one shared transport/auth client abstraction for both supervisor middleware and gateway interceptors rather than introducing two security-sensitive client implementations. Related roadmap: #1904.

Technical Context

Today both extension systems create tonic channels independently. Gateway interceptors accept HTTP, HTTPS, or Unix endpoints but do not configure TLS trust or attach caller credentials. Remote supervisor middleware accepts HTTP or HTTPS; HTTPS uses platform roots only, and the same unauthenticated registration is used by the gateway for Describe/ValidateConfig and by sandbox supervisors for Describe/EvaluateHttpRequest.

OpenShell already has most of the underlying mechanics: the gateway loads one Ed25519 signing key and kid, SandboxJwtIssuer creates short-lived tokens, RefreshSandboxToken renews a sandbox's in-memory bearer credential, and AuthInterceptor replaces bearer metadata in place. The missing work is a distinct extension audience/claim contract, authorization of requested extension credentials, per-service token slots, a shared extension channel builder, and custom-CA configuration and propagation.

Affected Components

Component	Key Files	Role
Shared extension core	new crates/openshell-extension-core/, crates/openshell-core/src/config.rs	Extension-wide primitives shared by middleware and interceptors, including endpoint validation, TLS/UDS channels, bearer injection, and per-service token slots
Gateway token authority	crates/openshell-server/src/auth/sandbox_jwt.rs, crates/openshell-server/src/lib.rs	Reuse the configured Ed25519 key, kid, issuer, and TTL to mint distinct extension-audience JWTs
Supervisor token distribution	proto/openshell.proto, crates/openshell-server/src/grpc/auth_rpc.rs, crates/openshell-core/src/grpc_client.rs	Extend the existing sandbox-only refresh API to request and return authorized per-service credentials
Middleware registration	crates/openshell-server/src/config_file.rs, proto/sandbox.proto, crates/openshell-supervisor-middleware/src/lib.rs	Hold stable audience/TLS configuration and distribute public CA PEM to supervisors
Middleware client	crates/openshell-supervisor-middleware/src/remote.rs	Consume the shared channel and bearer-slot implementation for all remote RPCs
Interceptor client	crates/openshell-gateway-interceptors/src/plan.rs	Consume the same shared channel and bearer-slot implementation while preserving Unix sockets
Sandbox reconciliation	crates/openshell-sandbox/src/lib.rs	Acquire credentials before remote registry connection and update slots without reconnecting on token rotation
JWKS publication	crates/openshell-server/src/http.rs, crates/openshell-server/src/multiplex.rs	Expose the currently configured public key and kid as a single-key JWKS for integration verification

Technical Investigation

Architecture Overview

The gateway constructs the middleware registry at startup from [[openshell.supervisor.middleware]], calls every remote service's Describe, and later calls ValidateConfig during policy admission. MiddlewareRegistry::required_services copies only policy-selected registrations into GetSandboxConfigResponse. Every sandbox supervisor receives those registrations over its already-authenticated gateway channel and constructs the same remote registry for request evaluation.

Gateway interceptors follow a separate startup path: ExecutionPlan::load connects each endpoint, calls Describe, then stores cloned tonic clients for the configured bindings and optional provider-profile source. The interceptor and middleware crates both depend on openshell-core, but neither has a neutral reusable extension transport layer.

The gateway currently loads the JWT issuer after it has already connected middleware and gateway interceptors. Authenticated startup calls therefore require moving signing-material construction earlier and passing an extension token source into both client initializers.

Stable connection data and rotating credentials must remain separate:

• Custom CA PEM is public, stable trust material. It can be loaded once by the gateway, included in SupervisorMiddlewareService, hashed into config_revision, and trigger a registry rebuild when it changes.
• Bearer JWTs are secret and short-lived. They must not be placed in SupervisorMiddlewareService: the gateway hashes the full protobuf message into config_revision, and supervisors compare complete service vectors to decide whether to rebuild and reconnect the registry.

Code References

Location	Description
crates/openshell-server/src/config_file.rs:196-229	Middleware TOML currently contains only name, endpoint, body limit, and timeout, then maps infallibly to the delivered proto.
crates/openshell-core/src/config.rs:617-649	Interceptor configuration has no shared auth, audience, or custom-CA fields.
proto/sandbox.proto:343-388	GetSandboxConfigResponse delivers stable middleware registrations; SupervisorMiddlewareService has no CA or auth identity.
crates/openshell-server/src/grpc/policy.rs:1592-1637	The gateway resolves the middleware services required by the effective sandbox policy and returns them to the supervisor.
crates/openshell-server/src/grpc/policy.rs:3941-3990	The full encoded middleware registration participates in config_revision.
crates/openshell-supervisor-middleware/src/lib.rs:667-780	Gateway and supervisor callers both construct external services through MiddlewareRegistry::connect_services.
crates/openshell-supervisor-middleware/src/lib.rs:821-839	required_services selects registrations referenced by effective policy; this is the authorization source for supervisor extension-token requests.
crates/openshell-supervisor-middleware/src/remote.rs:25-56	Middleware HTTPS currently uses platform roots only and the generated client has no bearer interceptor.
crates/openshell-gateway-interceptors/src/plan.rs:175-212	Interceptor startup connects and calls Describe without caller credentials.
crates/openshell-gateway-interceptors/src/plan.rs:860-892	Interceptor network and Unix channel construction is local to the interceptor crate.
crates/openshell-server/src/auth/sandbox_jwt.rs:36-126	Existing Ed25519 issuer hard-codes sandbox-to-gateway claims and audience but provides reusable key, kid, issuer, TTL, and encoding mechanics.
proto/openshell.proto:559-580	IssueSandboxToken and RefreshSandboxToken are existing sandbox-authenticated token APIs.
proto/openshell.proto:651-679	Existing token responses are secret-marked and explicitly held in supervisor memory.
crates/openshell-server/src/grpc/auth_rpc.rs:108-155	Refresh rejects non-sandbox principals and renews a still-valid gateway JWT.
crates/openshell-core/src/grpc_client.rs:43-123	Existing refreshable bearer slot and tonic interceptor replace request metadata without rebuilding the channel.
crates/openshell-core/src/grpc_client.rs:324-403	Existing renewal loop refreshes at approximately 80% of remaining lifetime and keeps the current credential during transient failures.
crates/openshell-sandbox/src/lib.rs:2035-2095	Supervisor startup installs built-ins first, then connects delivered remote middleware with retry/degraded behavior.
crates/openshell-sandbox/src/lib.rs:2308-2338	Complete registration equality controls middleware registry reconciliation.
crates/openshell-sdk/src/transport.rs:129-143	Existing SDK custom-CA behavior provides a TLS configuration pattern to follow.
crates/openshell-server/src/lib.rs:261-279,321-326,354-403	Middleware/interceptors initialize before the JWT issuer is loaded, so startup composition must be reordered.

Current Behavior

1. The gateway materializes middleware registrations directly from TOML and connects them before constructing ServerState.
2. RemoteMiddlewareService::connect enables native roots for HTTPS and otherwise opens an unauthenticated channel.
3. Interceptors call their own connect_endpoint, including a private Unix connector, and store unauthenticated generated clients.
4. The gateway later loads the sandbox JWT key pair and constructs a fixed-audience SandboxJwtIssuer.
5. Supervisors authenticate to the gateway with a process-global bearer slot and call GetSandboxConfig every ten seconds. Middleware registration changes rebuild the remote registry.
6. RefreshSandboxToken renews only the sandbox-to-gateway credential. It has no request fields and returns no extension credentials.

What Would Need to Change

Shared extension core

Create openshell-extension-core as the neutral foundation consumed by both extension systems. It should own resolved endpoint validation, HTTP/HTTPS/Unix channel construction, platform or custom-CA trust, normal hostname verification, shared extension identity and claim types, bearer metadata injection, expiry-aware per-service token slots, and secret-safe Debug/error behavior. It must not depend on middleware- or interceptor-specific generated clients and must preserve conditional Unix support.

The crate boundary follows one rule: it contains protocol-neutral primitives shared by at least two extension mechanisms; subsystem-specific orchestration stays with its owning crate. Keep operator-facing and wire configuration in existing ownership layers (openshell-core config types, server TOML, and protobufs) and keep signing authority in openshell-server. This gives the dependency direction gateway-interceptors|supervisor-middleware -> openshell-extension-core without a cycle. Cargo workspace membership is automatic through crates/*; Bazel needs the corresponding crate target and dependency metadata. Document this ownership rule in the new crate README to prevent core from becoming a general extension dumping ground.

Extension token contract and minting

Add a distinct extension claim set while reusing the existing gateway signing material and encoding mechanics. The alpha claim contract should include exact audience, issuer, subject, issued-at, expiry, kid, unique token ID, caller kind (gateway or supervisor), and sandbox ID only for supervisor calls. The existing sandbox JWT and its gateway audience must remain unchanged and must never be sent to an extension.

Each interceptor and middleware registration needs a stable operator-owned extension audience or a documented deterministic derivation. Gateway-originated clients mint the same claim form locally with gateway caller identity. A single-key JWKS representation of the already-loaded public key and kid gives integrations a verification mechanism without requiring multi-key rotation in the alpha slice.

Reuse RefreshSandboxToken for distribution

Extend RefreshSandboxTokenRequest additively with requested extension registration names, not arbitrary audiences. Extend the response with secret-marked service name, token, and expiry tuples while preserving the existing gateway token fields.

The handler must resolve every requested name against server-owned registration metadata and the effective policy for the authenticated sandbox before minting. Unknown, unselected, or duplicate requests fail rather than letting the caller choose an audience. IssueSandboxToken remains the Kubernetes bootstrap exchange because required middleware is not known until after the first configuration fetch.

Supervisors learn stable service names from GetSandboxConfig, acquire their credentials over the already-authenticated cached gateway client, then connect or reconcile the registry. Subsequent refresh updates per-service slots in memory without changing stable service equality or reconnecting channels. A transient refresh failure may retain the current credential until expiry; extension calls must fail locally after expiry.

Custom CA

Add the same custom-CA path field to middleware and interceptor operator configuration. The gateway reads and validates the file during startup. Interceptors pass the bytes directly to the shared connector; middleware registration carries the public PEM bytes through SupervisorMiddlewareService, so gateway validation and supervisor evaluation use the same trust material without driver-specific mounts.

Configured custom trust should preserve ordinary certificate and endpoint-hostname verification. Following the current SDK pattern, the proposed alpha semantics are that a configured CA replaces platform roots (private-CA pinning), while omission uses platform roots. A CA supplied for HTTP or Unix transport is invalid configuration.

Initialization and reconciliation

Construct the reusable signer/token source before middleware and interceptor startup. Remote clients receive bearer slots rather than raw token strings. Middleware reconciliation must match stable service identity separately from rotating credentials, update existing slots atomically, acquire a new credential before connecting an added service, and remove slots for detached services.

Alternative Approaches Considered

Use a transport-only openshell-extension-transport crate

This name precisely describes the immediate channel, TLS, and bearer work, but becomes artificially narrow once shared extension identity, claim types, registration primitives, or future verification helpers need a neutral owner. openshell-extension-core is selected so those cross-mechanism primitives have a durable home, guarded by the explicit shared-by-two-mechanisms ownership rule.

Put rotating tokens in GetSandboxConfig

This reuses the configuration poll but mixes short-lived secret state with a response that is also user-callable. Putting token bytes inside the registration would additionally change config_revision and force registry reconnection on rotation. Conditional out-of-band fields could be made safe, but the existing sandbox-only refresh RPC is a better semantic and authorization boundary.

Add a new extension-token RPC

A dedicated RPC would produce a clean contract, but RefreshSandboxToken already authenticates the supervisor, renews secret in-memory material, and is used by every compute-driver path. Additive request/response fields provide the alpha mechanism with less API and client duplication.

Use static bearer files

Static files are simpler for gateway callers but require separate secret mounts for Kubernetes, Docker, Podman, and VM supervisors and create a second integration contract that would later be replaced. The existing authenticated refresh API avoids those driver changes.

Reuse the sandbox-to-gateway JWT

Rejected. Its audience authorizes gateway access and it represents a different trust boundary. Giving it to extension services would expose a gateway credential and eliminate exact service audience separation.

Send custom-CA paths to supervisors

Rejected. Gateway host paths do not map consistently into four sandbox runtimes. CA certificates are public; delivering resolved PEM through stable configuration avoids new mounts, file labels, and driver-specific behavior.

Patterns to Follow

• Preserve the existing SandboxJwtIssuer Ed25519 algorithm pinning, kid, issuer identity, configured TTL, and secret-safe diagnostics.
• Follow AuthInterceptor's shared in-memory bearer replacement, but use independent slots per extension service rather than the process-global gateway slot.
• Follow RefreshSandboxToken's sandbox-principal and still-valid-token authorization boundary.
• Follow openshell-sdk::transport for operator-provided CA construction and normal hostname verification.
• Mark every new token protobuf field with (openshell.options.v1.secret) = true and extend the descriptor test that verifies dedicated secret fields.
• Preserve middleware's last-known-good registry behavior and existing on_error policy semantics.
• Use existing private-CA/mTLS test helpers under crates/openshell-server/tests/common/ and the remote tonic-server test pattern in openshell-supervisor-middleware.

Proposed Approach

Introduce openshell-extension-core as the shared foundation for middleware and interceptor clients, with transport as its first internal module and an explicit cross-mechanism ownership rule. Reuse the gateway's configured Ed25519 key and token encoder to mint a separate extension claim contract, then extend RefreshSandboxToken to return only the per-service credentials authorized by the caller sandbox's effective policy. Keep stable endpoint, audience, and public CA data in configuration, but keep rotating bearer credentials in per-service memory slots outside config revision and registry equality. Publish the current public key as a single-key JWKS for alpha integrations. Defer mTLS, multi-key overlap rotation, hot key reload, HA rotation coordination, and driver-specific secret delivery to later hardening.

Alpha Scope and Deferred Hardening

In scope

• One openshell-extension-core implementation of shared transport/auth primitives used by middleware and interceptors.
• Exact-audience gateway-minted bearer JWTs for gateway and supervisor callers.
• Supervisor distribution through additive RefreshSandboxToken fields.
• Single-key JWKS verification material.
• Platform roots or operator-provided custom CA with normal hostname validation.
• Gateway-to-interceptor, gateway-to-middleware, and supervisor-to-middleware calls.
• Explicit failure on missing, unauthorized, malformed, or expired credentials.
• Targeted interoperability, TLS, authorization, reconciliation, and secret-isolation tests.

Deferred

• mTLS client authentication.
• Multi-key overlap rotation and hot signing-key reload.
• HA issuer/key rollout semantics.
• Per-runtime secret mounts or certificate delivery.
• End-user delegation or durable agent identity.
• Runtime extension registration and health management.
• Unix socket ownership/TOCTOU hardening beyond preserving current support.

Scope Assessment

• Complexity: Medium
• Confidence: High — existing minting, refresh, polling, and TLS patterns provide a clear path
• Estimated files to change: 14-20 source, test, documentation, and build files
• Estimated agent-assisted effort: 5-8 focused engineering days
• Issue type: feat

Risks & Open Questions

• Confirm the exact audience syntax: an explicit stable URI is clearer for integrations; a deterministic subsystem/name derivation reduces configuration.
• Confirm that a configured custom CA replaces platform roots, matching the current SDK, rather than augmenting them.
• Decide whether the alpha token TTL reuses gateway_jwt.ttl_secs or adds a separately bounded extension-token TTL.
• Define whether one invalid requested service rejects the entire credential batch or returns valid entries plus per-entry errors; atomic failure is simpler and safer.
• Decide whether RefreshSandboxToken should mint a fresh gateway token when the request only needs extension credentials; preserving current behavior is simplest.
• A public JWKS route needs a stable path and must not expose private material. Multi-key semantics remain intentionally deferred.
• Gateway startup order must change so signing material exists before authenticated extension Describe; startup errors must not print tokens or PEM contents.
• openshell-extension-core needs an enforced ownership rule and README because a broadly named core crate can otherwise accumulate middleware- or interceptor-specific orchestration.

Disposition Readiness

• State: state:validated
• Assessment: The current code paths, reusable primitives, authorization boundary, shared-client ownership, CA propagation, and deferred hardening are sufficiently understood for a human accept/decline decision.
• Missing evidence: None. The listed audience, CA trust-set, and TTL choices are contract decisions for disposition or implementation planning, not missing feasibility evidence.

Test Considerations

• Shared TLS tests must run a real tonic server signed by a private CA and prove success with the configured CA, plus failure with an absent/wrong CA and hostname mismatch.
• Verify both extension systems use the shared channel path and attach bearer metadata to Describe, ValidateConfig, interceptor evaluation, middleware evaluation, and profile snapshot calls where applicable.
• JWT unit tests must cover gateway versus supervisor claims, exact audience, unique jti, expiry, kid, and rejection of a sandbox-to-gateway token for an extension audience.
• Refresh authorization tests must prove a sandbox can request only middleware selected by its effective policy and cannot supply an arbitrary audience.
• Proto/config tests must cover CA-path reading, malformed or unreadable files, PEM propagation, secret annotations, and backwards-compatible empty new fields.
• Sandbox tests must prove token replacement updates a slot without changing config_revision or rebuilding the middleware registry, and expired credentials fail locally into the existing middleware on_error handling.
• Secret-hygiene tests must prove tokens are absent from Debug, errors, logs, policy/settings/provider environments, sidecar control, files, and child process environments.
• Run the affected crate tests plus sandbox e2e because supervisor transport and reconciliation change. No new driver-specific e2e is required when CA PEM and tokens are distributed through existing authenticated gateway APIs.
• LSM impact: No direct SELinux/AppArmor-specific behavior is expected. The gateway performs an ordinary configured-file read, while PEM and tokens travel in memory over existing authenticated gRPC. No new /proc, process identity, executable, file-label, or cross-domain mount behavior is introduced.

Documentation Impact

• Update docs/reference/gateway-config.mdx for the new shared audience/custom-CA/insecure fields and changed extension transport behavior.
• Update docs/extensibility/supervisor-middleware.mdx and docs/extensibility/gateway-interceptors.mdx with the shared JWT verification contract and CA examples.
• Update RFC 0009's authentication/delivery discussion and RFC 0010's currently deferred auth section.
• Update the relevant existing architecture gateway/sandbox overview rather than creating a new top-level architecture file.
• Update Helm values/templates/docs only if first-class chart-managed CA mounting is included; inline CA propagation to supervisors itself requires no compute-driver or LSM documentation changes.

---

Created by spike investigation. state:validated means the issue is ready for human disposition; state:needs-info means specific evidence is still required. A human applies state:accepted if OpenShell should pursue the work and places it on the roadmap separately. To queue unattended agent planning, a human applies agent:plan-requested; a direct request to an agent does not require that label.

[pimlock]

🔒 security-review-agent

Security Review

Determination: Legitimate concern

Summary

Remote middleware and gateway-interceptor services currently have no cryptographic way to distinguish OpenShell gateway or sandbox-supervisor RPCs from direct network callers. TLS authenticates the extension service to OpenShell, but without caller authentication a reachable extension can be invoked as a confused deputy or fed forged evaluation/interception requests.

Severity Assessment

• Impact: High when an extension endpoint is network-reachable or performs privileged transformations/actions; medium when trusted network isolation is enforced.
• Exploitability: Moderate. Exploitation requires reachability to the extension endpoint, but does not require compromise of the gateway signing key.
• Affected components: openshell-supervisor-middleware, openshell-gateway-interceptors, gateway extension-client initialization, sandbox token refresh, supervisor credential handling, gateway JWKS publication, and extension TLS/custom-CA configuration.

Attack Scenario

1. An attacker discovers or gains network access to a middleware or interceptor endpoint.
2. The attacker directly invokes its RPCs while impersonating an OpenShell gateway or supervisor; the service cannot distinguish the call from legitimate OpenShell traffic.
3. The service processes attacker-controlled input, potentially exposing request data, performing privileged integration actions, or returning crafted decisions that weaken enforcement.

Remediation Plan

1. Add openshell-extension-core and consume its shared endpoint, TLS, and bearer-token primitives from both middleware and interceptor clients. Authenticate every remote extension RPC with short-lived EdDSA JWTs minted by the existing gateway signing authority. Use a distinct extension claim contract with exact service audience, issuer, kid, expiry, unique jti, caller kind, and sandbox identity where applicable.
2. Extend the sandbox-only RefreshSandboxToken request/response additively for per-service credentials. Accept only registered service names, derive audiences from server-owned registration data, authorize them against the authenticated sandbox's effective policy, and reject unknown or unselected services. Never accept a caller-provided audience.
3. Keep rotating credentials in secret-safe per-service memory slots, outside policy, configuration revision, environment variables, files, sidecars, logs, and child processes. Fail closed when a credential is absent or expired, without reconnecting channels on routine refresh.
4. Publish the existing public key and kid as a single-key JWKS, while documenting operator-provisioned public-key bootstrap for extension services that cannot reach the gateway. Treat authenticated TLS or an operator-established channel—not the public JWKS payload itself—as the trust root.
5. Add shared HTTPS custom-CA support for middleware and interceptor connections with normal hostname verification. Reject CA configuration for plaintext HTTP and Unix sockets, and propagate public middleware CA PEM through existing authenticated sandbox configuration instead of runtime-specific mounts.
6. Add negative tests for audience substitution, caller-kind mismatch, sandbox-token confusion, unauthorized refresh requests, unknown kid, expiration, secret leakage, wrong CA, and hostname mismatch. Verify token refresh does not alter config_revision or rebuild middleware registries.

Additional Notes

The alpha mechanism intentionally retains residual risk: bearer tokens can be replayed until their short expiry, and single-key JWKS publication makes coordinated key replacement operationally brittle. Exact audiences, short TTLs, authenticated TLS, policy-constrained refresh, and memory-only token handling are reasonable alpha mitigations; multi-key overlap, mTLS, and stronger replay resistance remain deferred hardening.