feat: handle Kubernetes sandbox pod eviction and recovery after infrastructure disruptions

[varshaprasad96]

Problem Statement

Kubernetes sandbox pods can be evicted or terminated for several infrastructure reasons, including:

• Planned node maintenance and drains.
• Cluster autoscaling.
• Node upgrades or graceful shutdown.
• Resource-pressure eviction.
• Scheduler preemption.
• Node failure or loss.

When this happens, OpenShell must determine:

• Whether to protect the pod from a planned eviction.
• Whether there is time to checkpoint state.
• Whether a replacement pod was created.
• Whether the replacement can run on another node.
• Which sandbox state survived.
• How to communicate disruption and recovery to users.

OpenShell currently reduces these situations to broad lifecycle phases such as Provisioning. It does not explain that the sandbox was evicted, identify the affected pod instance or node, show why recovery is blocked, or indicate whether durable state was restored.

Node-Drain Experiment

To investigate this behavior, we deployed OpenShell into a two-worker kind cluster and created a sandbox through the OpenShell CLI. The gateway ran on one worker and the sandbox ran on the other.

We then drained the worker hosting the sandbox.

The experiment showed:

• A normal drain refused to evict the sandbox because the pod used an emptyDir.
• Drain succeeded after adding --delete-emptydir-data.
• The OpenShell sandbox record and Agent Sandbox CR were preserved.
• Agent Sandbox created a replacement pod with the same name and a new UID.
• The replacement could not move to the second worker because kind's local-path PV was bound to the original node.
• OpenShell displayed only Provisioning while recovery was blocked.
• After the original node was uncordoned, the replacement became Ready using the same PVC.
• Files under /sandbox survived.
• Files under /tmp, running processes, exec sessions, and connections did not survive.

This experiment demonstrates one planned infrastructure disruption. The solution should also address unplanned disruptions, recognizing that checkpointing is only possible when OpenShell receives advance notice.

Proposed Design

1. Surface disruption and recovery

Add durable sandbox conditions such as:

Disrupted=True
Reason=PodEvicted

Recovering=True
Reason=Unschedulable

Expose:

• Current and previous node.
• Pod or runtime-instance UID.
• Replacement status.
• Scheduling or storage failure.
• Disruption time.
• Recovery duration.
• Whether durable state was restored.

Update the API, CLI, JSON/YAML output, TUI, logs, and metrics to distinguish initial provisioning from recovery.

2. Protect planned evictions

Support an optional per-sandbox PodDisruptionBudget.

Testing confirmed that the following behavior blocks voluntary eviction:

minAvailable: 1

Because a sandbox is a singleton, the replacement cannot be created until the original pod is deleted. The PDB must therefore be explicitly relaxed after checkpointing or by an operator.

A PDB protects against voluntary Eviction API operations. It cannot prevent direct pod deletion, sudden node failure, or drains that bypass eviction.

3. Checkpoint before planned eviction

For planned maintenance:

1. Cordon the node.
2. Keep the PDB active.
3. Mark the sandbox as Checkpointing.
4. Stop accepting new exec and SSH sessions.
5. Drain or interrupt active operations.
6. Invoke an optional application checkpoint hook.
7. Flush durable state into /sandbox.
8. Record checkpoint completion in the gateway.
9. Relax the PDB.
10. Evict the pod and observe recovery.

Example configuration concepts:

disruption:
  evictionProtection: true
  checkpointGracePeriodSeconds: 90
  terminationGracePeriodSeconds: 120
  checkpointFailurePolicy: Block

The checkpoint deadline should be shorter than the Kubernetes termination grace period, leaving time for process shutdown and supervisor cleanup.

Potential checkpoint failure policies include:

• Block: retain eviction protection and require operator intervention.
• Proceed: allow eviction even if checkpointing fails.
• ProceedAfterTimeout: block initially and proceed when the deadline expires.

Application-level checkpointing should be the initial scope. It can preserve:

• Files under /sandbox.
• Git working state.
• Agent task metadata.
• Application-specific checkpoints.
• Information needed to restart interrupted work.

Transparent restoration of process memory, sockets, exec sessions, and SSH sessions is out of scope.

A Kubernetes preStop hook or SIGTERM handler could provide a simpler best-effort checkpoint. However, eviction has already been accepted at that point and cannot be cancelled if checkpointing fails. See Kubernetes lifecycle hooks.

For unplanned disruption, OpenShell should recover whatever is available from persistent storage and explicitly report that no checkpoint occurred.

4. Document storage durability requirements

Document that:

• OpenShell's default /sandbox PVC survives replacement of the pod while the Sandbox CR remains.
• Cross-node recovery requires a portable CSI-backed StorageClass.
• kind's default local-path storage remains tied to its original node.
• emptyDir, /tmp, container-layer changes, and process state do not survive.
• Externally managed PVCs can make data ownership independent of the Sandbox CR.
• Retain reclaim policies can protect storage after PVC or sandbox deletion.
• CSI VolumeSnapshots can provide additional recovery points.
• --delete-emptydir-data is required for the current sandbox pod layout during drain.

Alternatives Considered

PodDisruptionBudget only

A PDB successfully blocks voluntary eviction, but it does not checkpoint state or provide availability. Because the sandbox is a singleton, the replacement cannot be created until the original pod is deleted.

Longer termination grace period only

This gives the supervisor more time to shut down, but eviction has already started. It cannot block deletion when checkpointing fails.

Pod finalizers

Finalizers can leave pod objects stuck in Terminating, but they do not keep the workload usable if its node is unavailable. They are not suitable as an eviction-protection mechanism.

Portable storage only

Portable storage allows the replacement to move to another node and preserves /sandbox, but it does not preserve processes, connections, or active sessions.

Full process checkpoint and restore

CRIU or runtime-level checkpointing could theoretically preserve process memory. Compatibility across kernels, container runtimes, namespaces, sockets, mounts, and security configurations makes this unsuitable for the initial design.

Agent Investigation

Environment

• OpenShell revision: 8c7dd148
• Built images: v0.0.90-67-g8c7dd148
• Agent Sandbox controller: v0.5.0
• Kubernetes: v1.31.0
• Cluster: kind with two worker nodes
• StorageClass: standard
• Provisioner: rancher.io/local-path

Observed identities

Resource	Before drain	After drain
OpenShell sandbox ID	2e3ae382-2272-4229-90bc-c5a8bd60c017	Unchanged
Agent Sandbox CR UID	e81b7fb9-669b-4ae0-a56b-e4b39027174c	Unchanged
Pod UID	45d3c0d0-3910-461f-a4c8-72fff452417b	0d2b3a54-0936-485b-b027-afbb64897347
PVC UID	5ba67215-0d8f-462f-b3ed-d0440ed33bea	Unchanged
OpenShell phase	Ready	Provisioning, then Ready

State validation

• /sandbox/pvc-marker: survived.
• /tmp/ephemeral-marker: lost.
• The replacement remained Pending until the original node was uncordoned.
• OpenShell exposed only Provisioning while Kubernetes showed a Pending replacement.

PDB validation

• minAvailable: 1
• expectedPods: 1
• desiredHealthy: 1
• disruptionsAllowed: 0
• Drain timed out after repeated PDB violations.
• The pod UID remained unchanged.

Relevant implementation findings

• The Kubernetes driver creates an Agent Sandbox CR and default workspace PVC.
• It watches Sandbox CRs and events but does not retain pod UID or node identity.
• CLI and TUI primarily expose the normalized phase.
• Kubernetes driver configuration does not expose lifecycle hooks or termination grace periods.
• The supervisor handles SIGTERM by forwarding it to the entrypoint and waiting for it to exit.
• There is currently no checkpoint protocol or checkpoint acknowledgement.

Acceptance Criteria

• OpenShell distinguishes initial provisioning from disruption recovery.
• Sandbox status exposes node and runtime-instance identity.
• Eviction, replacement, scheduling failure, and recovery are visible through the API, CLI, TUI, logs, and metrics.
• Optional per-sandbox PDB protection blocks voluntary eviction.
• A checkpoint workflow has explicit success, timeout, and failure behavior.
• Replacement pods report whether durable state was restored.
• Documentation explains persistent, ephemeral, and storage-topology behavior.
• Multi-worker Kubernetes tests cover eviction, PDB blocking, local-storage failure, portable-storage recovery, and checkpoint timeout.

[varshaprasad96]

Here are also findings from the evicting pod that runs gateway:

• Deleting the gateway pod did not restart or reschedule the sandbox pod.
• Sandbox ID, Kubernetes Sandbox CR, pod UID, workspace PVC, files, /tmp contents, entrypoint, and background processes remained unchanged.
• The replacement gateway loaded its existing SQLite database from the gateway PVC and reconciled sandboxes with the Kubernetes driver.
• The sandbox briefly appeared as Provisioning.
• Its supervisor automatically reconnected to the replacement gateway in approximately six seconds, after which the sandbox returned to Ready.
• New exec requests worked after reconnection.
• The test validated gateway-pod replacement only. The gateway’s SQLite PVC is node-local, so gateway-node failure could prevent the replacement pod from starting.
• Durable or external database storage is required for reliable gateway failover. Checkpoint artifacts should likewise not be stored in the gateway pod.

Overall, gateway-pod failure causes temporary control-plane and session disruption, but does not affect the running sandbox when gateway storage persists. Active sessions are the main state lost.