Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .agents/skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -355,6 +355,23 @@ helm -n openshell get values openshell | grep sandboxNamespace

Then inspect sandbox resources in that namespace.

If `workspace_namespaces` is present in `gateway.toml`, inspect every mapped
namespace rather than only `sandboxNamespace`:

```bash
kubectl -n openshell get configmap openshell-config \
-o jsonpath='{.data.gateway\.toml}' | grep workspace_namespaces
kubectl -n <mapped-namespace> get sandbox,pod,event
kubectl auth can-i list sandboxes.agents.x-k8s.io \
--namespace <mapped-namespace> \
--as system:serviceaccount:openshell:openshell
```

An unmapped workspace is rejected before the driver creates a Sandbox CR.
Registration failures from an otherwise healthy mapped sandbox usually mean
the gateway RoleBinding, sandbox ServiceAccount name, or projected-token
bootstrap prerequisites differ between namespaces.

Check the configured sandbox service account when TokenReview bootstrap or
sandbox registration fails. Helm creates a dedicated sandbox service account by
default and writes it to `[openshell.drivers.kubernetes].service_account_name`;
Expand Down
8 changes: 8 additions & 0 deletions architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,14 @@ through the driver configuration. The Helm chart defaults sandbox agents to
`Unconfined` so runtime/default AppArmor profiles do not block supervisor
network namespace setup on AppArmor-enabled nodes.

The Kubernetes driver can map logical workspaces to a bounded set of
pre-provisioned namespaces. Creation resolves the namespace from the workspace;
all other lifecycle operations reconcile across the distinct configured
namespace set. Kubernetes ServiceAccount bootstrap authentication uses the same
allowlist before reading a pod or its owning Sandbox CR. The driver never
creates or deletes namespaces, and a non-empty mapping rejects unmapped
workspaces rather than falling back to a shared namespace.

Resource requirements enter the driver layer through `SandboxSpec.resource_requirements`. This includes a set of GPU requirements, where a user
can request a specific number of GPUs or the driver-specific default behaviour.
For all in-tree drivers, this is equivalent to selecting a single GPU.
Expand Down
18 changes: 16 additions & 2 deletions crates/openshell-driver-kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@
Kubernetes-backed compute driver for OpenShell cluster deployments.

The driver uses the Kubernetes API to create, delete, fetch, and watch sandbox
custom resources in the configured namespace. It runs in-process with the
gateway server.
custom resources. By default it uses one configured namespace. Operators can
instead configure `workspace_namespaces` to map logical OpenShell workspaces to
pre-provisioned Kubernetes namespaces. It runs in-process with the gateway
server.

## Runtime Model

Expand All @@ -26,6 +28,18 @@ by the gateway.
Kubernetes API calls use explicit timeouts so gRPC handlers do not block
indefinitely when the API server is slow or unavailable.

## Workspace Namespace Mapping

An empty `workspace_namespaces` map preserves legacy behavior and routes every
sandbox to `namespace`. A non-empty map is an allowlist: sandbox creation fails
when its logical workspace is not mapped. Get, list, delete, existence checks,
Sandbox watches, Event watches, OpenShift SCC discovery, and ServiceAccount
bootstrap authentication operate across only the distinct mapped namespaces.

The driver does not create namespaces or RBAC. Operators must install the
sandbox ServiceAccount, Secrets, Role, and gateway RoleBinding in every mapped
namespace. Mapping changes require a gateway restart.

## Workspace Persistence

Sandbox pods use a PVC-backed `/sandbox` workspace. An init container seeds the
Expand Down
133 changes: 131 additions & 2 deletions crates/openshell-driver-kubernetes/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

use openshell_core::config;
use serde::{Deserialize, Deserializer, Serialize};
use std::collections::{BTreeMap, BTreeSet};
use std::path::Path;
use std::str::FromStr;

Expand Down Expand Up @@ -233,6 +234,11 @@ where
#[serde(default, deny_unknown_fields)]
pub struct KubernetesComputeConfig {
pub namespace: String,
/// Optional mapping from logical `OpenShell` workspace names to
/// pre-provisioned Kubernetes namespaces. When empty, all sandboxes use
/// `namespace` for backward compatibility. When non-empty, sandbox
/// creation fails closed for an unmapped workspace.
pub workspace_namespaces: BTreeMap<String, String>,
/// Kubernetes `ServiceAccount` assigned to sandbox pods and accepted by
/// the gateway's `TokenReview` bootstrap authenticator.
pub service_account_name: String,
Expand Down Expand Up @@ -333,6 +339,7 @@ impl Default for KubernetesComputeConfig {
fn default() -> Self {
Self {
namespace: DEFAULT_K8S_NAMESPACE.to_string(),
workspace_namespaces: BTreeMap::new(),
service_account_name: DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME.to_string(),
default_image: openshell_core::image::default_sandbox_image(),
// Default empty so the gateway omits `imagePullPolicy` from pod
Expand Down Expand Up @@ -364,6 +371,55 @@ impl Default for KubernetesComputeConfig {
}

impl KubernetesComputeConfig {
/// Resolve the Kubernetes namespace for a logical workspace.
///
/// An empty mapping retains the legacy single-namespace behavior. Once a
/// mapping is configured, every workspace must be explicitly present.
pub fn namespace_for_workspace(&self, workspace: &str) -> Result<&str, String> {
if self.workspace_namespaces.is_empty() {
return Ok(&self.namespace);
}
self.workspace_namespaces
.get(workspace)
.map(String::as_str)
.ok_or_else(|| {
format!("workspace '{workspace}' is not mapped to a Kubernetes namespace")
})
}

/// Namespaces the driver is allowed to manage, in deterministic order.
#[must_use]
pub fn sandbox_namespaces(&self) -> Vec<&str> {
if self.workspace_namespaces.is_empty() {
return vec![self.namespace.as_str()];
}
self.workspace_namespaces
.values()
.map(String::as_str)
.collect::<BTreeSet<_>>()
.into_iter()
.collect()
}

/// Validate the legacy namespace and every configured workspace mapping.
pub fn validate_workspace_namespaces(&self) -> Result<(), String> {
validate_kubernetes_namespace_name(&self.namespace)?;
for (workspace, namespace) in &self.workspace_namespaces {
if workspace.trim().is_empty() {
return Err("workspace_namespaces keys must not be empty".to_string());
}
if workspace.trim() != workspace {
return Err(format!(
"workspace_namespaces key '{workspace}' must not contain leading or trailing whitespace"
));
}
validate_kubernetes_namespace_name(namespace).map_err(|err| {
format!("invalid namespace mapped from workspace '{workspace}': {err}")
})?;
}
Ok(())
}

/// Clamp `sa_token_ttl_secs` into the `[MIN_SA_TOKEN_TTL_SECS,
/// MAX_SA_TOKEN_TTL_SECS]` range used by the projected-volume spec.
/// Invalid (≤0) values fall back to the default 3600.
Expand Down Expand Up @@ -405,7 +461,7 @@ impl KubernetesComputeConfig {
/// 3. Fallback defaults: UID=`1000`, GID=UID
pub fn resolve_sandbox_uid(
&self,
namespace_annotations: Option<&std::collections::BTreeMap<String, String>>,
namespace_annotations: Option<&BTreeMap<String, String>>,
) -> u32 {
if let Some(uid) = self.sandbox_uid {
return uid;
Expand All @@ -423,7 +479,7 @@ impl KubernetesComputeConfig {
pub fn resolve_sandbox_gid(
&self,
resolved_uid: u32,
_namespace_annotations: Option<&std::collections::BTreeMap<String, String>>,
_namespace_annotations: Option<&BTreeMap<String, String>>,
) -> u32 {
self.sandbox_gid
.or(self.sandbox_uid)
Expand Down Expand Up @@ -475,6 +531,24 @@ impl KubernetesComputeConfig {
}
}

fn validate_kubernetes_namespace_name(namespace: &str) -> Result<(), String> {
let bytes = namespace.as_bytes();
let valid = !bytes.is_empty()
&& bytes.len() <= 63
&& bytes.first().is_some_and(u8::is_ascii_alphanumeric)
&& bytes.last().is_some_and(u8::is_ascii_alphanumeric)
&& bytes
.iter()
.all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || *byte == b'-');
if valid {
Ok(())
} else {
Err(format!(
"Kubernetes namespace '{namespace}' must be a lowercase DNS-1123 label of at most 63 characters"
))
}
}

fn validate_provider_spiffe_workload_api_socket_path_value(
socket_path: &str,
) -> Result<(), String> {
Expand Down Expand Up @@ -527,6 +601,61 @@ mod tests {
assert!(cfg.workspace_storage_class.is_empty());
}

#[test]
fn empty_workspace_mapping_preserves_legacy_namespace() {
let cfg = KubernetesComputeConfig {
namespace: "shared".to_string(),
..KubernetesComputeConfig::default()
};
assert_eq!(cfg.namespace_for_workspace("default").unwrap(), "shared");
assert_eq!(cfg.sandbox_namespaces(), vec!["shared"]);
}

#[test]
fn workspace_mapping_resolves_and_fails_closed() {
let cfg = KubernetesComputeConfig {
workspace_namespaces: BTreeMap::from([
("team-a".to_string(), "app-a".to_string()),
("team-b".to_string(), "app-b".to_string()),
]),
..KubernetesComputeConfig::default()
};
cfg.validate_workspace_namespaces().unwrap();
assert_eq!(cfg.namespace_for_workspace("team-a").unwrap(), "app-a");
assert!(
cfg.namespace_for_workspace("unmapped")
.unwrap_err()
.contains("not mapped")
);
assert_eq!(cfg.sandbox_namespaces(), vec!["app-a", "app-b"]);
}

#[test]
fn workspace_mapping_deduplicates_shared_target_namespaces() {
let cfg = KubernetesComputeConfig {
workspace_namespaces: BTreeMap::from([
("team-a".to_string(), "shared-app".to_string()),
("team-b".to_string(), "shared-app".to_string()),
]),
..KubernetesComputeConfig::default()
};
assert_eq!(cfg.sandbox_namespaces(), vec!["shared-app"]);
}

#[test]
fn workspace_mapping_rejects_invalid_namespaces_and_keys() {
for workspace_namespaces in [
BTreeMap::from([(String::new(), "app-a".to_string())]),
BTreeMap::from([("team-a".to_string(), "Invalid_Namespace".to_string())]),
] {
let cfg = KubernetesComputeConfig {
workspace_namespaces,
..KubernetesComputeConfig::default()
};
assert!(cfg.validate_workspace_namespaces().is_err());
}
}

#[test]
fn default_topology_is_combined() {
let cfg = KubernetesComputeConfig::default();
Expand Down
Loading