diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index e1e731a0ce..5ccbcaadd2 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -308,6 +308,18 @@ Resource requirements enter the driver layer through `SandboxSpec.resource_requi can request a specific number of GPUs or the driver-specific default behaviour. For all in-tree drivers, this is equivalent to selecting a single GPU. +For Docker GPU sandboxes, the driver treats CDI specs as runtime metadata for +both outer injection and inner sandbox policy. It selects opaque CDI device IDs, +passes them to Docker, mounts daemon-reported CDI spec directories into +supervisor-only paths, and bind-mounts a gateway-owned versioned CDI context +read-only before creating the container. The supervisor resolves that context +inside the sandbox and derives Landlock paths and supplemental groups from CDI +`containerEdits`. Host-side CDI spec paths are diagnostic only and are never +treated as sandbox policy paths. +Kubernetes must not infer CDI device IDs from the `nvidia.com/gpu` resource +request; it needs a node-local selected-device handoff before using the same +supervisor resolver. + VM runtime state paths are derived only from driver-validated sandbox IDs matching `[A-Za-z0-9._-]{1,128}`. The gateway-owned VM driver socket uses a private `run/` directory plus Unix peer UID/PID checks. Standalone diff --git a/crates/openshell-core/src/cdi.rs b/crates/openshell-core/src/cdi.rs index 66350a01bc..d71059d352 100644 --- a/crates/openshell-core/src/cdi.rs +++ b/crates/openshell-core/src/cdi.rs @@ -9,12 +9,21 @@ use serde::{Deserialize, Serialize}; pub const CDI_CONTEXT_VERSION: u32 = 1; +/// File name used for the serialized CDI context. +pub const CDI_CONTEXT_FILE_NAME: &str = "cdi-context.json"; + /// Absolute supervisor path for the CDI context file mounted by a compute driver. pub const CDI_CONTEXT_PATH: &str = "/run/openshell/supervisor/cdi-context.json"; /// Base supervisor path under which compute drivers mount CDI specification directories. pub const CDI_SPEC_DIR_BASE: &str = "/run/openshell/supervisor/cdi-specs"; +/// Return the supervisor path used for a CDI specification directory. +#[must_use] +pub fn cdi_spec_mount_path(index: usize) -> String { + format!("{CDI_SPEC_DIR_BASE}/{index}") +} + #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct CdiContext { pub version: u32, diff --git a/crates/openshell-driver-docker/README.md b/crates/openshell-driver-docker/README.md index bbd7e69b88..a7b1640ae4 100644 --- a/crates/openshell-driver-docker/README.md +++ b/crates/openshell-driver-docker/README.md @@ -107,9 +107,35 @@ contract: | `PidsLimit` | Enforces the sandbox PID budget at the Docker cgroup layer. Set `[openshell.drivers.docker].sandbox_pids_limit = 0` to inherit the Docker/runtime default. | | CDI GPU request | Uses opaque `driver_config.cdi_devices` values when set; otherwise selects the requested count of NVIDIA CDI GPUs in round-robin order when daemon CDI support is detected. Docker daemon `/info` can permit `nvidia.com/gpu=all` as a WSL2 all-only compatibility fallback, where it counts as one selectable device. Exact CDI device lists must not contain duplicates and must match the effective GPU count. | | `policy-dns-transparent-tcp` capability | Declares that the combined Docker supervisor can own namespace-local DNS/TCP capture and coupled workload restart. The shared supervisor still owns DNS eligibility, mappings, authorization, pinned dialing, relaying, and OCSF decisions. The marker is stripped from the workload environment. | +| CDI context mount | For GPU/CDI sandboxes only, creates a gateway-owned context file and bind-mounts it read-only at `/run/openshell/supervisor/cdi-context.json`; daemon-reported CDI spec directories are mounted read-only under `/run/openshell/supervisor/cdi-specs/`. | The agent child process does not retain these supervisor privileges. +## CDI GPU Metadata + +Docker remains the source of truth for GPU injection. The driver selects opaque +CDI device IDs from `driver_config.cdi_devices` or the daemon's discovered CDI +inventory, then passes the same IDs to Docker with a CDI `DeviceRequest`. + +When a GPU/CDI request is present, the driver also mounts the Docker +daemon-reported `Info.CDISpecDirs` into supervisor-only paths. Before container +creation, it writes a small versioned CDI context in gateway-owned state and +bind-mounts it read-only into the supervisor. The context uses container-side +spec paths for resolution and keeps host-side spec sources diagnostic-only. If +context or token creation fails, the driver removes any created state files; if +container creation or start fails, it also removes the container and state +files before reporting the failure. + +The sandbox supervisor resolves the selected IDs from those mounted specs +before it launches agent processes. CDI device nodes become read-write +Landlock paths, mount destinations default to read-only paths, and +`additionalGids` become supplemental groups for the entrypoint and SSH child +processes. Writable CDI mount destinations are accepted only for exact +single-file paths already listed in the sandbox policy `read_write` list; +writable CDI directory mounts fail closed. Kubernetes, Podman, WSL2 hardware +validation, and Tegra/Jetson hardware validation are separate follow-up +targets. + ## Driver Config Mounts The gateway forwards the `docker` block from `--driver-config-json` to this diff --git a/crates/openshell-driver-docker/src/lib.rs b/crates/openshell-driver-docker/src/lib.rs index 4566d5c171..33f0015ecc 100644 --- a/crates/openshell-driver-docker/src/lib.rs +++ b/crates/openshell-driver-docker/src/lib.rs @@ -21,6 +21,7 @@ use bollard::query_parameters::{ }; use bytes::Bytes; use futures::{Stream, StreamExt}; +use openshell_core::cdi::{CdiContext, CdiSpecDirectory, cdi_spec_mount_path}; use openshell_core::config::{DEFAULT_SANDBOX_PIDS_LIMIT, DEFAULT_STOP_TIMEOUT_SECS}; use openshell_core::driver_mounts; use openshell_core::driver_utils::{ @@ -210,17 +211,64 @@ struct DockerDriverRuntimeConfig { supervisor_bin: PathBuf, guest_tls: Option, daemon_version: String, - gpu: DockerGpuRuntimeCapabilities, + gpu: DockerGpuRuntimeConfig, sandbox_pids_limit: i64, enable_bind_mounts: bool, } -#[derive(Debug, Clone, Copy)] -struct DockerGpuRuntimeCapabilities { - cdi_supported: bool, +#[derive(Debug, Clone, Default)] +struct DockerGpuRuntimeConfig { + cdi_spec_dirs: Vec, wsl_all_gpu_fallback_enabled: bool, } +impl DockerGpuRuntimeConfig { + fn supports_gpu(&self) -> bool { + !self.cdi_spec_dirs.is_empty() + } + + fn cdi_context(&self, gpu_device_ids: Option<&[String]>) -> Result, Status> { + let Some(gpu_device_ids) = gpu_device_ids.filter(|device_ids| !device_ids.is_empty()) + else { + return Ok(None); + }; + self.require_cdi_spec_dirs()?; + Ok(Some(CdiContext::new( + gpu_device_ids.to_vec(), + self.cdi_spec_dirs + .iter() + .enumerate() + .map(|(index, source)| CdiSpecDirectory::new(cdi_spec_mount_path(index), source)) + .collect(), + ))) + } + + fn cdi_spec_bind_strings( + &self, + gpu_device_ids: Option<&[String]>, + ) -> Result, Status> { + let Some(_) = gpu_device_ids.filter(|device_ids| !device_ids.is_empty()) else { + return Ok(Vec::new()); + }; + self.require_cdi_spec_dirs()?; + Ok(self + .cdi_spec_dirs + .iter() + .enumerate() + .map(|(index, source)| format!("{source}:{}:ro,z", cdi_spec_mount_path(index))) + .collect()) + } + + fn require_cdi_spec_dirs(&self) -> Result<(), Status> { + if self.cdi_spec_dirs.is_empty() { + return Err(Status::failed_precondition( + "docker GPU sandboxes require Docker CDI spec directories reported by the daemon", + )); + } + Ok(()) +} +} + #[derive(Debug, Clone, PartialEq, Eq)] enum DockerGatewayRoute { Bridge { @@ -504,16 +552,11 @@ impl DockerComputeDriver { let info = docker.info().await.map_err(|err| { Error::execution(format!("failed to query Docker daemon info: {err}")) })?; - let cdi_supported = info - .cdi_spec_dirs - .as_ref() - .is_some_and(|dirs| !dirs.is_empty()); - let cdi_gpu_inventory = docker_cdi_gpu_inventory(&info); - let wsl_all_gpu_fallback_enabled = docker_info_reports_wsl2(&info); - let gpu = DockerGpuRuntimeCapabilities { - cdi_supported, - wsl_all_gpu_fallback_enabled, + let gpu = DockerGpuRuntimeConfig { + cdi_spec_dirs: info.cdi_spec_dirs.clone().unwrap_or_default(), + wsl_all_gpu_fallback_enabled: docker_info_reports_wsl2(&info), }; + let cdi_gpu_inventory = docker_cdi_gpu_inventory(&info); validate_sandbox_pids_limit(docker_config.sandbox_pids_limit)?; let gateway_port = gateway_bind_address.port(); if gateway_port == 0 { @@ -563,7 +606,7 @@ impl DockerComputeDriver { supervisor_bin, guest_tls, daemon_version: version.version.unwrap_or_else(|| "unknown".to_string()), - gpu, + gpu: gpu.clone(), sandbox_pids_limit: docker_config.sandbox_pids_limit, enable_bind_mounts: docker_config.enable_bind_mounts, }, @@ -600,8 +643,8 @@ impl DockerComputeDriver { limit_supported: true, }), gpu: Some(GpuResourceCapabilities { - default_selection_supported: self.config.gpu.cdi_supported, - count_selection_supported: self.config.gpu.cdi_supported, + default_selection_supported: self.config.gpu.supports_gpu(), + count_selection_supported: self.config.gpu.supports_gpu(), }), }), rootfs_tar_staging_dir: String::new(), @@ -637,7 +680,7 @@ impl DockerComputeDriver { DockerSandboxDriverConfig::from_template(template).map_err(Status::invalid_argument)?; validate_docker_driver_mounts(&driver_config.mounts, config.enable_bind_mounts)?; let gpu_requirements = driver_gpu_requirements(spec.resource_requirements.as_ref()); - Self::validate_gpu_request(gpu_requirements, config.gpu.cdi_supported, &driver_config)?; + Self::validate_gpu_request(gpu_requirements, config.gpu.supports_gpu(), &driver_config)?; Ok(ValidatedDockerSandbox { template, driver_config, @@ -935,12 +978,6 @@ impl DockerComputeDriver { image.ref = %template.image, )) .await?; - let token_file_created = write_sandbox_token_file(sandbox, &self.config) - .await - .map_err(|status| { - DockerProvisioningFailure::new("SandboxTokenWriteFailed", status.message()) - })?; - let container_name = container_name_for_sandbox(sandbox); let gpu_devices = self .resolve_gpu_cdi_devices( @@ -950,9 +987,13 @@ impl DockerComputeDriver { ) .await .map_err(|status| { - if token_file_created { - cleanup_sandbox_token_file(sandbox, &self.config); - } + DockerProvisioningFailure::new("ContainerCreateFailed", status.message()) + })?; + let cdi_context = self + .config + .gpu + .cdi_context(gpu_devices.as_deref()) + .map_err(|status| { DockerProvisioningFailure::new("ContainerCreateFailed", status.message()) })?; let create_body = build_container_create_body_for_image( @@ -963,11 +1004,24 @@ impl DockerComputeDriver { &image, ) .map_err(|status| { - if token_file_created { - cleanup_sandbox_token_file(sandbox, &self.config); - } DockerProvisioningFailure::new("ContainerCreateFailed", status.message()) })?; + if let Some(cdi_context) = cdi_context.as_ref() + && let Err(status) = write_cdi_context_file(sandbox, &self.config, cdi_context) + { + cleanup_cdi_context_file(sandbox, &self.config); + return Err(DockerProvisioningFailure::new( + "CdiContextWriteFailed", + status.message(), + )); + } + if let Err(status) = write_sandbox_token_file(sandbox, &self.config).await { + cleanup_cdi_context_file(sandbox, &self.config); + return Err(DockerProvisioningFailure::new( + "SandboxTokenWriteFailed", + status.message(), + )); + } async { openshell_otel::record_error_result( self.docker @@ -981,9 +1035,7 @@ impl DockerComputeDriver { ) .await .map_err(|err| { - if token_file_created { - cleanup_sandbox_token_file(sandbox, &self.config); - } + cleanup_sandbox_state_files(sandbox, &self.config); DockerProvisioningFailure::from_status( "ContainerCreateFailed", create_status_from_docker_error("create docker sandbox container", err), @@ -1020,24 +1072,13 @@ impl DockerComputeDriver { )) .await; if let Err(err) = start_result { - let cleanup = self - .docker - .remove_container( - &container_name, - Some(RemoveContainerOptionsBuilder::default().force(true).build()), - ) - .await; - if let Err(cleanup_err) = cleanup { - warn!( - sandbox_id = %sandbox.id, - container_name, - error = %cleanup_err, - "Failed to clean up Docker container after start failure" - ); - } - if token_file_created { - cleanup_sandbox_token_file(sandbox, &self.config); - } + self.cleanup_created_container_after_failure( + &sandbox.id, + &container_name, + "container start failure", + ) + .await; + cleanup_sandbox_state_files(sandbox, &self.config); return Err(DockerProvisioningFailure::from_status( "ContainerStartFailed", create_status_from_docker_error("start docker sandbox container", err), @@ -1063,6 +1104,30 @@ impl DockerComputeDriver { span_status.finish(Ok(())) } + async fn cleanup_created_container_after_failure( + &self, + sandbox_id: &str, + container_name: &str, + phase: &'static str, + ) { + let cleanup = self + .docker + .remove_container( + container_name, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ) + .await; + if let Err(cleanup_err) = cleanup { + warn!( + sandbox_id = %sandbox_id, + container_name = %container_name, + phase, + error = %cleanup_err, + "Failed to clean up Docker container after provisioning failure" + ); + } + } + async fn delete_sandbox_inner( &self, sandbox_id: &str, @@ -1090,11 +1155,11 @@ impl DockerComputeDriver { .await { Ok(()) => { - cleanup_sandbox_token_file(&record.sandbox, &self.config); + cleanup_sandbox_state_files(&record.sandbox, &self.config); return Ok(true); } Err(err) if is_not_found_error(&err) => { - cleanup_sandbox_token_file(&record.sandbox, &self.config); + cleanup_sandbox_state_files(&record.sandbox, &self.config); return Ok(true); } Err(err) => { @@ -1117,11 +1182,11 @@ impl DockerComputeDriver { .await { Ok(()) => { - cleanup_sandbox_token_file_for_delete(sandbox_id, pending.as_ref(), &self.config); + cleanup_sandbox_state_files_for_delete(sandbox_id, pending.as_ref(), &self.config); Ok(true) } Err(err) if is_not_found_error(&err) => { - cleanup_sandbox_token_file_for_delete(sandbox_id, pending.as_ref(), &self.config); + cleanup_sandbox_state_files_for_delete(sandbox_id, pending.as_ref(), &self.config); Ok(pending.is_some()) } Err(err) => Err(internal_status("delete docker sandbox container", err)), @@ -1137,7 +1202,7 @@ impl DockerComputeDriver { if let Some(task) = record.task { task.abort(); } - cleanup_sandbox_token_file(&record.sandbox, &self.config); + cleanup_sandbox_state_files(&record.sandbox, &self.config); self.publish_deleted(record.sandbox.id); return Ok(()); } @@ -1314,7 +1379,7 @@ impl DockerComputeDriver { sandbox: &DriverSandbox, failure: &DockerProvisioningFailure, ) { - cleanup_sandbox_token_file(sandbox, &self.config); + cleanup_sandbox_state_files(sandbox, &self.config); let snapshot = pending_sandbox_snapshot( sandbox, &self.config.sandbox_namespace, @@ -2612,6 +2677,7 @@ fn docker_volume_is_bind_backed(volume: &bollard::models::Volume) -> bool { fn build_binds( sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig, + gpu_device_ids: Option<&[String]>, ) -> Result, Status> { let mut binds = vec![format!( "{}:{}:ro,z", @@ -2638,6 +2704,13 @@ fn build_binds( SANDBOX_TOKEN_MOUNT_PATH )); } + if cdi_context_requested(gpu_device_ids) { + binds.push(format!( + "{}:{}:ro,z", + cdi_context_host_path(sandbox, config)?.display(), + openshell_core::cdi::CDI_CONTEXT_PATH + )); + } Ok(binds) } @@ -2664,6 +2737,57 @@ fn sandbox_token_host_path_by_id( }) } +fn cdi_context_host_path( + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, +) -> Result { + cdi_context_host_path_by_id(&sandbox.id, config) +} + +fn cdi_context_host_path_by_id( + sandbox_id: &str, + config: &DockerDriverRuntimeConfig, +) -> Result { + openshell_core::driver_utils::sandbox_token_path( + "docker-cdi-contexts", + Some(&config.sandbox_namespace), + sandbox_id, + ) + .map(|path| path.with_file_name(openshell_core::cdi::CDI_CONTEXT_FILE_NAME)) + .map_err(|err| Status::internal(format!("resolve CDI context state directory failed: {err}"))) +} + +fn write_cdi_context_file( + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, + context: &CdiContext, +) -> Result<(), Status> { + let path = cdi_context_host_path(sandbox, config)?; + if let Some(parent) = path.parent() { + openshell_core::paths::create_dir_restricted(parent).map_err(|err| { + Status::internal(format!( + "create CDI context directory {} failed: {err}", + parent.display() + )) + })?; + } + let json = serde_json::to_vec(context) + .map_err(|err| Status::internal(format!("encode CDI context failed: {err}")))?; + std::fs::write(&path, json).map_err(|err| { + Status::internal(format!( + "write CDI context file {} failed: {err}", + path.display() + )) + })?; + openshell_core::paths::set_file_owner_only(&path).map_err(|err| { + Status::internal(format!( + "restrict CDI context file {} failed: {err}", + path.display() + )) + })?; + Ok(()) +} + async fn write_sandbox_token_file( sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig, @@ -2704,6 +2828,15 @@ fn cleanup_sandbox_token_file(sandbox: &DriverSandbox, config: &DockerDriverRunt cleanup_sandbox_token_file_by_id(&sandbox.id, config); } +fn cleanup_cdi_context_file(sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig) { + cleanup_cdi_context_file_by_id(&sandbox.id, config); +} + +fn cleanup_sandbox_state_files(sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig) { + cleanup_sandbox_token_file(sandbox, config); + cleanup_cdi_context_file(sandbox, config); +} + fn cleanup_sandbox_token_file_for_delete( sandbox_id: &str, pending: Option<&PendingSandboxRecord>, @@ -2716,6 +2849,27 @@ fn cleanup_sandbox_token_file_for_delete( } } +fn cleanup_cdi_context_file_for_delete( + sandbox_id: &str, + pending: Option<&PendingSandboxRecord>, + config: &DockerDriverRuntimeConfig, +) { + if !sandbox_id.is_empty() { + cleanup_cdi_context_file_by_id(sandbox_id, config); + } else if let Some(record) = pending { + cleanup_cdi_context_file(&record.sandbox, config); + } +} + +fn cleanup_sandbox_state_files_for_delete( + sandbox_id: &str, + pending: Option<&PendingSandboxRecord>, + config: &DockerDriverRuntimeConfig, +) { + cleanup_sandbox_token_file_for_delete(sandbox_id, pending, config); + cleanup_cdi_context_file_for_delete(sandbox_id, pending, config); +} + fn cleanup_sandbox_token_file_by_id(sandbox_id: &str, config: &DockerDriverRuntimeConfig) { let Ok(path) = sandbox_token_host_path_by_id(sandbox_id, config) else { return; @@ -2735,15 +2889,39 @@ fn cleanup_sandbox_token_file_by_id(sandbox_id: &str, config: &DockerDriverRunti } } +fn cleanup_cdi_context_file_by_id(sandbox_id: &str, config: &DockerDriverRuntimeConfig) { + let Ok(path) = cdi_context_host_path_by_id(sandbox_id, config) else { + return; + }; + if let Err(err) = std::fs::remove_file(&path) + && err.kind() != std::io::ErrorKind::NotFound + { + warn!( + sandbox_id = %sandbox_id, + path = %path.display(), + error = %err, + "Failed to remove Docker CDI context file" + ); + } + if let Some(dir) = path.parent() { + let _ = std::fs::remove_dir(dir); + } +} + #[cfg(test)] -fn build_environment(sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig) -> Vec { - build_environment_for_oci_user(sandbox, config, "") +fn build_environment( + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, + include_cdi_context: bool, +) -> Vec { + build_environment_for_oci_user(sandbox, config, "", include_cdi_context) } fn build_environment_for_oci_user( sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig, oci_user: &str, + include_cdi_context: bool, ) -> Vec { let mut environment = HashMap::from([ ("HOME".to_string(), "/root".to_string()), @@ -2803,6 +2981,14 @@ fn build_environment_for_oci_user( openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES.to_string(), openshell_core::sandbox_env::POLICY_DNS_TRANSPARENT_TCP_CAPABILITY.to_string(), ); + environment.insert( + openshell_core::sandbox_env::CDI_CONTEXT.to_string(), + if include_cdi_context { + openshell_core::cdi::CDI_CONTEXT_PATH.to_string() + } else { + String::new() + }, + ); // The root supervisor executes namespace helpers during bootstrap; keep // their search path driver-owned even when the template/spec set PATH. environment.insert("PATH".to_string(), SUPERVISOR_PATH.to_string()); @@ -2890,6 +3076,10 @@ fn docker_gpu_selection_status(err: CdiGpuSelectionError) -> Status { Status::failed_precondition(err.to_string()) } +fn cdi_context_requested(gpu_device_ids: Option<&[String]>) -> bool { + gpu_device_ids.is_some_and(|device_ids| !device_ids.is_empty()) +} + #[cfg(test)] fn build_container_create_body( sandbox: &DriverSandbox, @@ -3026,7 +3216,12 @@ fn build_container_create_body_for_image( // The image workspace may need to be created or rejected by the // supervisor, so do not let the OCI runtime chdir there first. working_dir: Some("/".to_string()), - env: Some(build_environment_for_oci_user(sandbox, config, &image.user)), + env: Some(build_environment_for_oci_user( + sandbox, + config, + &image.user, + cdi_context_requested(gpu_device_ids), + )), entrypoint: Some(vec![SUPERVISOR_MOUNT_PATH.to_string()]), // Replace the image CMD with the supervisor's resolved workspace // argument so Docker cannot append inherited image arguments. @@ -3038,7 +3233,8 @@ fn build_container_create_body_for_image( pids_limit: docker_pids_limit(config.sandbox_pids_limit)?, device_requests, binds: { - let mut binds = build_binds(sandbox, config)?; + let mut binds = build_binds(sandbox, config, gpu_device_ids)?; + binds.extend(config.gpu.cdi_spec_bind_strings(gpu_device_ids)?); binds.extend(user_bind_strings); Some(binds) }, diff --git a/crates/openshell-driver-docker/src/tests.rs b/crates/openshell-driver-docker/src/tests.rs index 010d50c20e..6aa534dbb9 100644 --- a/crates/openshell-driver-docker/src/tests.rs +++ b/crates/openshell-driver-docker/src/tests.rs @@ -93,7 +93,10 @@ fn gpu_resources(count: Option) -> ResourceRequirements { } } -fn runtime_config() -> DockerDriverRuntimeConfig { +const TEST_CDI_SPEC_DIR: &str = "/opt/openshell-test/cdi"; +const TEST_CDI_SPEC_DIR_ALT: &str = "/srv/openshell-test/cdi"; + +fn runtime_config(supports_gpu: bool) -> DockerDriverRuntimeConfig { DockerDriverRuntimeConfig { default_image: "image:latest".to_string(), image_pull_policy: String::new(), @@ -121,15 +124,32 @@ fn runtime_config() -> DockerDriverRuntimeConfig { key: PathBuf::from("/tmp/tls.key"), }), daemon_version: "28.0.0".to_string(), - gpu: DockerGpuRuntimeCapabilities { - cdi_supported: false, - wsl_all_gpu_fallback_enabled: false, - }, + gpu: gpu_runtime_config(supports_gpu), sandbox_pids_limit: DEFAULT_SANDBOX_PIDS_LIMIT, enable_bind_mounts: false, } } +fn runtime_config_with_cdi_spec_dirs(cdi_spec_dirs: &[&str]) -> DockerDriverRuntimeConfig { + let mut config = runtime_config(false); + config.gpu.cdi_spec_dirs = cdi_spec_dirs + .iter() + .map(|path| (*path).to_string()) + .collect(); + config +} + +fn gpu_runtime_config(supports_gpu: bool) -> DockerGpuRuntimeConfig { + if supports_gpu { + DockerGpuRuntimeConfig { + cdi_spec_dirs: vec![TEST_CDI_SPEC_DIR.to_string()], + ..Default::default() + } + } else { + DockerGpuRuntimeConfig::default() + } +} + fn json_struct(value: serde_json::Value) -> prost_types::Struct { let serde_json::Value::Object(object) = value else { panic!("expected JSON object"); @@ -173,7 +193,7 @@ fn test_driver_with_config(config: DockerDriverRuntimeConfig) -> DockerComputeDr #[test] fn capabilities_report_static_resource_support() { - let mut config = runtime_config(); + let mut config = runtime_config(false); let capabilities = test_driver_with_config(config.clone()).capabilities(); let resources = capabilities.resource_capabilities.unwrap(); assert!(resources.cpu.unwrap().limit_supported); @@ -182,7 +202,7 @@ fn capabilities_report_static_resource_support() { assert!(!gpu.default_selection_supported); assert!(!gpu.count_selection_supported); - config.gpu.cdi_supported = true; + config.gpu.cdi_spec_dirs = vec![TEST_CDI_SPEC_DIR.to_string()]; let gpu = test_driver_with_config(config) .capabilities() .resource_capabilities @@ -219,7 +239,7 @@ async fn standalone_traced_client() -> ( let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); let address = listener.local_addr().unwrap(); let (shutdown, shutdown_rx) = tokio::sync::oneshot::channel(); - let service = ComputeDriverService::new(test_driver_with_config(runtime_config())); + let service = ComputeDriverService::new(test_driver_with_config(runtime_config(false))); let server = tokio::spawn(async move { tonic::transport::Server::builder() .layer(openshell_otel::compute_driver_rpc_layer()) @@ -327,7 +347,8 @@ async fn tracing_in_process_service_preserves_the_driver_rpc_server_boundary() { otel_tracing::TRACING.in_process_targets(), )) .with(otel_tracing::TRACING.in_process_layer(&driver_provider)); - let service = ComputeDriverService::new_in_process(test_driver_with_config(runtime_config())); + let service = + ComputeDriverService::new_in_process(test_driver_with_config(runtime_config(false))); async { let gateway_span = tracing::info_span!( @@ -467,7 +488,7 @@ async fn tracing_lifecycle_rpc_failures_export_docker_operation_spans() { .with_simple_exporter(exporter.clone()) .build(); let subscriber = tracing_subscriber::registry().with(otel_tracing::TRACING.layer(&provider)); - let driver = test_driver_with_config(runtime_config()); + let driver = test_driver_with_config(runtime_config(false)); async { ComputeDriver::create_sandbox( @@ -521,7 +542,7 @@ async fn tracing_direct_start_exports_a_docker_start_span() { .with_simple_exporter(exporter.clone()) .build(); let subscriber = tracing_subscriber::registry().with(otel_tracing::TRACING.layer(&provider)); - let driver = test_driver_with_config(runtime_config()); + let driver = test_driver_with_config(runtime_config(false)); DockerComputeDriver::start_sandbox(&driver, "", "") .with_subscriber(subscriber) @@ -553,7 +574,7 @@ async fn tracing_image_preparation_failure_exports_nested_failed_spans() { .with_simple_exporter(exporter.clone()) .build(); let subscriber = tracing_subscriber::registry().with(otel_tracing::TRACING.layer(&provider)); - let mut config = runtime_config(); + let mut config = runtime_config(false); config.image_pull_policy = "unsupported".to_string(); let driver = test_driver_with_config(config); @@ -786,7 +807,7 @@ async fn tracing_in_process_stream_leaves_status_unset_when_dropped() { #[tokio::test] async fn gateway_listener_requirements_report_managed_bridge_address() { - let config = runtime_config(); + let config = runtime_config(false); let expected_address = match config.gateway_route { DockerGatewayRoute::Bridge { bind_address, .. } => bind_address, DockerGatewayRoute::HostGateway => panic!("test config must use a managed bridge"), @@ -808,7 +829,7 @@ async fn gateway_listener_requirements_report_managed_bridge_address() { #[tokio::test] async fn gateway_listener_requirements_are_empty_for_host_gateway_route() { - let mut config = runtime_config(); + let mut config = runtime_config(false); config.gateway_route = DockerGatewayRoute::HostGateway; config.gateway_callback_bind_address = None; let driver = test_driver_with_config(config); @@ -824,7 +845,7 @@ async fn gateway_listener_requirements_are_empty_for_host_gateway_route() { #[tokio::test] async fn host_gateway_route_reports_ipv4_loopback_callback_listener() { - let mut config = runtime_config(); + let mut config = runtime_config(false); config.gateway_route = DockerGatewayRoute::HostGateway; config.gateway_callback_bind_address = Some("127.0.0.1:17670".parse().unwrap()); let driver = test_driver_with_config(config); @@ -1225,14 +1246,14 @@ fn docker_compute_config_disables_bind_mounts_by_default() { #[test] fn container_create_body_sets_driver_owned_pids_limit() { - let body = build_container_create_body(&test_sandbox(), &runtime_config()).unwrap(); + let body = build_container_create_body(&test_sandbox(), &runtime_config(false)).unwrap(); let host_config = body.host_config.expect("host config"); assert_eq!(host_config.pids_limit, Some(DEFAULT_SANDBOX_PIDS_LIMIT)); } #[test] fn build_environment_sets_docker_tls_paths() { - let env = build_environment(&test_sandbox(), &runtime_config()); + let env = build_environment(&test_sandbox(), &runtime_config(false), false); assert!(env.contains(&format!("OPENSHELL_TLS_CA={TLS_CA_MOUNT_PATH}"))); assert!(env.contains(&format!("OPENSHELL_TLS_CERT={TLS_CERT_MOUNT_PATH}"))); assert!(env.contains(&format!("OPENSHELL_TLS_KEY={TLS_KEY_MOUNT_PATH}"))); @@ -1265,7 +1286,7 @@ fn build_environment_keeps_network_capabilities_driver_controlled() { openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES.to_string(), "spoofed".to_string(), ); - let env = build_environment(&sandbox, &runtime_config()); + let env = build_environment(&sandbox, &runtime_config(false), false); assert!(env.contains(&format!( "{}={}", openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES, @@ -1286,7 +1307,7 @@ fn build_environment_protects_oci_identity_metadata() { spec.environment.insert(key.to_string(), value.to_string()); } - let env = build_environment_for_oci_user(&sandbox, &runtime_config(), "app:staff"); + let env = build_environment_for_oci_user(&sandbox, &runtime_config(false), "app:staff", false); assert!(env.contains(&format!( "{}=app:staff", @@ -1307,7 +1328,7 @@ fn build_environment_strips_gateway_tls_server_name() { "evil.attacker.example.com".to_string(), ); - let env = build_environment(&sandbox, &runtime_config()); + let env = build_environment(&sandbox, &runtime_config(false), false); assert!( !env.iter().any(|entry| entry.starts_with(&format!( @@ -1329,7 +1350,7 @@ fn container_creation_uses_inspected_immutable_image() { }; let body = build_container_create_body_for_image( &sandbox, - &runtime_config(), + &runtime_config(false), &DockerSandboxDriverConfig::default(), None, &metadata, @@ -1359,7 +1380,7 @@ fn container_creation_rejects_invalid_oci_working_dir() { }; let err = build_container_create_body_for_image( &test_sandbox(), - &runtime_config(), + &runtime_config(false), &DockerSandboxDriverConfig::default(), None, &metadata, @@ -1380,7 +1401,7 @@ fn container_creation_rejects_openshell_control_path_working_dir() { }; let err = build_container_create_body_for_image( &test_sandbox(), - &runtime_config(), + &runtime_config(false), &DockerSandboxDriverConfig::default(), None, &metadata, @@ -1403,7 +1424,7 @@ fn container_creation_rejects_image_volume_that_masks_working_dir() { let error = build_container_create_body_for_image( &sandbox, - &runtime_config(), + &runtime_config(false), &DockerSandboxDriverConfig::default(), None, &metadata, @@ -1425,7 +1446,7 @@ fn container_creation_rejects_image_volume_over_configured_ssh_socket() { working_dir: "/workspace".to_string(), volumes: vec!["/custom-runtime".to_string()], }; - let mut config = runtime_config(); + let mut config = runtime_config(false); config.ssh_socket_path = "/custom-runtime/ssh.sock".to_string(); let error = build_container_create_body_for_image( @@ -1454,7 +1475,7 @@ fn container_creation_reserves_resolved_workspace_root_but_allows_nested_mounts( .unwrap(); let err = build_container_create_body_for_image( &test_sandbox(), - &runtime_config(), + &runtime_config(false), &root_mount, None, &metadata, @@ -1476,7 +1497,7 @@ fn container_creation_reserves_resolved_workspace_root_but_allows_nested_mounts( }; let err = build_container_create_body_for_image( &test_sandbox(), - &runtime_config(), + &runtime_config(false), &ancestor_mount, None, &nested_metadata, @@ -1493,7 +1514,7 @@ fn container_creation_reserves_resolved_workspace_root_but_allows_nested_mounts( .unwrap(); build_container_create_body_for_image( &test_sandbox(), - &runtime_config(), + &runtime_config(false), &nested_mount, None, &metadata, @@ -1507,7 +1528,7 @@ fn container_creation_reserves_resolved_workspace_root_but_allows_nested_mounts( .unwrap(); build_container_create_body_for_image( &test_sandbox(), - &runtime_config(), + &runtime_config(false), &compatibility_path_mount, None, &metadata, @@ -1527,7 +1548,7 @@ fn build_environment_keeps_path_driver_controlled() { .environment .insert("PATH".to_string(), "/malicious/template/bin".to_string()); - let env = build_environment(&sandbox, &runtime_config()); + let env = build_environment(&sandbox, &runtime_config(false), false); let path_entries = env .iter() .filter(|entry| entry.starts_with("PATH=")) @@ -1553,7 +1574,7 @@ fn build_environment_keeps_telemetry_toggle_driver_controlled() { "true".to_string(), ); - let env = build_environment(&sandbox, &runtime_config()); + let env = build_environment(&sandbox, &runtime_config(false), false); let telemetry_entries = env .iter() .filter(|entry| { @@ -1575,7 +1596,7 @@ fn build_environment_keeps_telemetry_toggle_driver_controlled() { #[test] fn build_binds_uses_docker_tls_directory() { - let binds = build_binds(&test_sandbox(), &runtime_config()).unwrap(); + let binds = build_binds(&test_sandbox(), &runtime_config(false), None).unwrap(); let targets = binds .iter() .filter_map(|bind| bind.split(':').nth(1).map(String::from)) @@ -1614,7 +1635,7 @@ fn build_container_create_body_includes_driver_config_mounts() { ] }))); - let body = build_container_create_body(&sandbox, &runtime_config()).unwrap(); + let body = build_container_create_body(&sandbox, &runtime_config(false)).unwrap(); let mounts = body .host_config .unwrap() @@ -1662,7 +1683,7 @@ fn driver_config_defaults_volume_mounts_to_read_only() { }] }))); - let body = build_container_create_body(&sandbox, &runtime_config()).unwrap(); + let body = build_container_create_body(&sandbox, &runtime_config(false)).unwrap(); let mounts = body .host_config .unwrap() @@ -1691,7 +1712,7 @@ fn driver_config_allows_explicit_writable_volume_mounts() { }] }))); - let body = build_container_create_body(&sandbox, &runtime_config()).unwrap(); + let body = build_container_create_body(&sandbox, &runtime_config(false)).unwrap(); let mounts = body .host_config .unwrap() @@ -1725,7 +1746,7 @@ fn driver_config_rejects_duplicate_mount_targets() { ] }))); - let err = build_container_create_body(&sandbox, &runtime_config()).unwrap_err(); + let err = build_container_create_body(&sandbox, &runtime_config(false)).unwrap_err(); assert_eq!(err.code(), tonic::Code::FailedPrecondition); assert!( @@ -1752,7 +1773,7 @@ fn driver_config_rejects_bind_mounts_unless_enabled() { }] }))); - let err = build_container_create_body(&sandbox, &runtime_config()).unwrap_err(); + let err = build_container_create_body(&sandbox, &runtime_config(false)).unwrap_err(); assert_eq!(err.code(), tonic::Code::FailedPrecondition); assert!(err.message().contains("enable_bind_mounts = true")); @@ -1778,7 +1799,7 @@ fn build_container_create_body_includes_bind_mounts_when_enabled() { "read_only": true }] }))); - let mut config = runtime_config(); + let mut config = runtime_config(false); config.enable_bind_mounts = true; let body = build_container_create_body(&sandbox, &config).unwrap(); @@ -1823,7 +1844,7 @@ fn driver_config_defaults_enabled_bind_mounts_to_read_only() { "target": "/sandbox/host" }] }))); - let mut config = runtime_config(); + let mut config = runtime_config(false); config.enable_bind_mounts = true; let body = build_container_create_body(&sandbox, &config).unwrap(); @@ -1861,7 +1882,7 @@ fn bind_mount_selinux_shared_label() { "selinux_label": "shared" }] }))); - let mut config = runtime_config(); + let mut config = runtime_config(false); config.enable_bind_mounts = true; let body = build_container_create_body(&sandbox, &config).unwrap(); @@ -1899,7 +1920,7 @@ fn bind_mount_selinux_private_label() { "selinux_label": "private" }] }))); - let mut config = runtime_config(); + let mut config = runtime_config(false); config.enable_bind_mounts = true; let body = build_container_create_body(&sandbox, &config).unwrap(); @@ -1936,7 +1957,7 @@ fn bind_mount_without_selinux_label() { "read_only": false }] }))); - let mut config = runtime_config(); + let mut config = runtime_config(false); config.enable_bind_mounts = true; let body = build_container_create_body(&sandbox, &config).unwrap(); @@ -1970,7 +1991,7 @@ fn driver_config_rejects_missing_bind_source() { "target": "/sandbox/data" }] }))); - let mut config = runtime_config(); + let mut config = runtime_config(false); config.enable_bind_mounts = true; let err = build_container_create_body(&sandbox, &config).unwrap_err(); @@ -2000,7 +2021,7 @@ fn driver_config_rejects_relative_bind_sources_when_enabled() { "target": "/sandbox/host" }] }))); - let mut config = runtime_config(); + let mut config = runtime_config(false); config.enable_bind_mounts = true; let err = build_container_create_body(&sandbox, &config).unwrap_err(); @@ -2030,7 +2051,7 @@ fn driver_config_rejects_image_mounts() { }] }))); - let err = build_container_create_body(&sandbox, &runtime_config()).unwrap_err(); + let err = build_container_create_body(&sandbox, &runtime_config(false)).unwrap_err(); assert_eq!(err.code(), tonic::Code::FailedPrecondition); assert!(err.message().contains("invalid docker driver_config")); @@ -2054,7 +2075,7 @@ fn driver_config_rejects_reserved_mount_targets() { }] }))); - let err = build_container_create_body(&sandbox, &runtime_config()).unwrap_err(); + let err = build_container_create_body(&sandbox, &runtime_config(false)).unwrap_err(); assert_eq!(err.code(), tonic::Code::FailedPrecondition); assert!(err.message().contains("reserved OpenShell path")); @@ -2075,7 +2096,7 @@ fn driver_config_rejects_mount_over_configured_ssh_socket() { working_dir: "/workspace".to_string(), volumes: Vec::new(), }; - let mut config = runtime_config(); + let mut config = runtime_config(false); config.ssh_socket_path = "/custom-runtime/ssh.sock".to_string(); let error = build_container_create_body_for_image( @@ -2152,7 +2173,7 @@ fn build_environment_uses_token_file_without_raw_token_env() { "user-provided-token".to_string(), ); - let env = build_environment(&sandbox, &runtime_config()); + let env = build_environment(&sandbox, &runtime_config(false), false); assert!(!env.iter().any(|entry| { entry.starts_with(&format!("{}=", openshell_core::sandbox_env::SANDBOX_TOKEN)) @@ -2176,7 +2197,7 @@ fn managed_container_label_filters_include_gateway_namespace() { #[test] fn build_container_create_body_replaces_inherited_cmd_with_workspace_arg() { - let create_body = build_container_create_body(&test_sandbox(), &runtime_config()).unwrap(); + let create_body = build_container_create_body(&test_sandbox(), &runtime_config(false)).unwrap(); assert_eq!( create_body.entrypoint, @@ -2225,7 +2246,7 @@ fn build_container_create_body_replaces_inherited_cmd_with_workspace_arg() { #[test] fn validate_sandbox_rejects_gpu_when_cdi_unavailable() { - let config = runtime_config(); + let config = runtime_config(false); let mut sandbox = test_sandbox(); sandbox.spec.as_mut().unwrap().resource_requirements = Some(gpu_resources(None)); @@ -2237,7 +2258,7 @@ fn validate_sandbox_rejects_gpu_when_cdi_unavailable() { #[test] fn validate_sandbox_rejects_missing_gpu_support_before_request_shape() { - let config = runtime_config(); + let config = runtime_config(false); let mut sandbox = test_sandbox(); let spec = sandbox.spec.as_mut().unwrap(); spec.resource_requirements = Some(gpu_resources(Some(2))); @@ -2251,7 +2272,7 @@ fn validate_sandbox_rejects_missing_gpu_support_before_request_shape() { #[test] fn validate_sandbox_rejects_invalid_cdi_devices_before_gpu_capability() { - let config = runtime_config(); + let config = runtime_config(false); let mut sandbox = test_sandbox(); let spec = sandbox.spec.as_mut().unwrap(); spec.resource_requirements = Some(gpu_resources(None)); @@ -2266,7 +2287,7 @@ fn validate_sandbox_rejects_invalid_cdi_devices_before_gpu_capability() { #[test] fn validate_sandbox_rejects_unknown_driver_config_fields() { - let config = runtime_config(); + let config = runtime_config(false); let mut sandbox = test_sandbox(); let spec = sandbox.spec.as_mut().unwrap(); spec.resource_requirements = Some(gpu_resources(None)); @@ -2281,8 +2302,7 @@ fn validate_sandbox_rejects_unknown_driver_config_fields() { #[test] fn validate_sandbox_accepts_gpu_count_request_shape() { - let mut config = runtime_config(); - config.gpu.cdi_supported = true; + let config = runtime_config(true); let mut sandbox = test_sandbox(); sandbox.spec.as_mut().unwrap().resource_requirements = Some(gpu_resources(Some(2))); @@ -2292,8 +2312,7 @@ fn validate_sandbox_accepts_gpu_count_request_shape() { #[test] fn validate_sandbox_accepts_gpu_count_matching_cdi_devices() { - let mut config = runtime_config(); - config.gpu.cdi_supported = true; + let config = runtime_config(true); let mut sandbox = test_sandbox(); let spec = sandbox.spec.as_mut().unwrap(); spec.resource_requirements = Some(gpu_resources(Some(2))); @@ -2308,8 +2327,7 @@ fn validate_sandbox_accepts_gpu_count_matching_cdi_devices() { #[test] fn validate_sandbox_accepts_single_cdi_device_without_gpu_count() { - let mut config = runtime_config(); - config.gpu.cdi_supported = true; + let config = runtime_config(true); let mut sandbox = test_sandbox(); let spec = sandbox.spec.as_mut().unwrap(); spec.resource_requirements = Some(gpu_resources(None)); @@ -2321,8 +2339,7 @@ fn validate_sandbox_accepts_single_cdi_device_without_gpu_count() { #[test] fn validate_sandbox_rejects_multiple_cdi_devices_without_gpu_count() { - let mut config = runtime_config(); - config.gpu.cdi_supported = true; + let config = runtime_config(true); let mut sandbox = test_sandbox(); let spec = sandbox.spec.as_mut().unwrap(); spec.resource_requirements = Some(gpu_resources(None)); @@ -2342,8 +2359,7 @@ fn validate_sandbox_rejects_multiple_cdi_devices_without_gpu_count() { #[test] fn validate_sandbox_rejects_cdi_devices_without_gpu_request() { - let mut config = runtime_config(); - config.gpu.cdi_supported = true; + let config = runtime_config(true); let mut sandbox = test_sandbox(); sandbox .spec @@ -2362,8 +2378,7 @@ fn validate_sandbox_rejects_cdi_devices_without_gpu_request() { #[test] fn validate_sandbox_rejects_gpu_count_mismatched_cdi_devices() { - let mut config = runtime_config(); - config.gpu.cdi_supported = true; + let config = runtime_config(true); let mut sandbox = test_sandbox(); let spec = sandbox.spec.as_mut().unwrap(); spec.resource_requirements = Some(gpu_resources(Some(2))); @@ -2380,7 +2395,7 @@ fn validate_sandbox_rejects_gpu_count_mismatched_cdi_devices() { #[test] fn validate_sandbox_rejects_template_errors_before_device_config() { - let config = runtime_config(); + let config = runtime_config(false); let mut sandbox = test_sandbox(); let spec = sandbox.spec.as_mut().unwrap(); spec.resource_requirements = Some(gpu_resources(None)); @@ -2418,8 +2433,7 @@ fn validate_sandbox_auth_accepts_gateway_token() { #[test] fn build_container_create_body_maps_default_gpu_to_selected_cdi_device() { - let mut config = runtime_config(); - config.gpu.cdi_supported = true; + let config = runtime_config(true); let mut sandbox = test_sandbox(); sandbox.spec.as_mut().unwrap().resource_requirements = Some(gpu_resources(None)); @@ -2446,10 +2460,106 @@ fn build_container_create_body_maps_default_gpu_to_selected_cdi_device() { ); } +#[test] +fn build_container_create_body_adds_cdi_context_env_and_spec_mounts_for_gpu() { + let config = runtime_config_with_cdi_spec_dirs(&[TEST_CDI_SPEC_DIR, TEST_CDI_SPEC_DIR_ALT]); + let mut sandbox = test_sandbox(); + sandbox.spec.as_mut().unwrap().resource_requirements = Some(gpu_resources(None)); + + let driver_config = DockerSandboxDriverConfig::default(); + let gpu_devices = vec!["nvidia.com/gpu=1".to_string()]; + let create_body = build_container_create_body_with_gpu_devices( + &sandbox, + &config, + &driver_config, + Some(&gpu_devices), + ) + .unwrap(); + + let env = create_body.env.expect("env should be set"); + assert!(env.iter().any(|entry| { + entry + == &format!( + "{}={}", + openshell_core::sandbox_env::CDI_CONTEXT, + openshell_core::cdi::CDI_CONTEXT_PATH + ) + })); + + let binds = create_body + .host_config + .expect("host config") + .binds + .expect("binds should be set"); + assert!( + binds.iter().any(|bind| { + bind == &format!("{TEST_CDI_SPEC_DIR}:{}:ro,z", cdi_spec_mount_path(0)) + }) + ); + assert!(binds.iter().any(|bind| { + bind == &format!("{TEST_CDI_SPEC_DIR_ALT}:{}:ro,z", cdi_spec_mount_path(1)) + })); + assert!(binds.iter().any(|bind| { + bind == &format!( + "{}:{}:ro,z", + cdi_context_host_path(&sandbox, &config).unwrap().display(), + openshell_core::cdi::CDI_CONTEXT_PATH + ) + })); +} + +#[test] +fn build_container_create_body_clears_cdi_context_for_non_gpu() { + let mut config = runtime_config(false); + config.gpu.cdi_spec_dirs = vec![TEST_CDI_SPEC_DIR.to_string()]; + let create_body = build_container_create_body(&test_sandbox(), &config).unwrap(); + + let env = create_body.env.expect("env should be set"); + assert!( + env.iter() + .any(|entry| { entry == &format!("{}=", openshell_core::sandbox_env::CDI_CONTEXT) }) + ); + + let binds = create_body + .host_config + .expect("host config") + .binds + .expect("binds should be set"); + assert!( + !binds + .iter() + .any(|bind| bind.contains(openshell_core::cdi::CDI_SPEC_DIR_BASE)) + ); +} + +#[test] +fn write_cdi_context_file_materializes_owned_host_context() { + let _guard = ENV_LOCK.lock().unwrap(); + let state_dir = tempfile::tempdir().unwrap(); + let sandbox = test_sandbox(); + let config = runtime_config(true); + let context = CdiContext::new( + vec!["nvidia.com/gpu=0".to_string()], + vec![CdiSpecDirectory::new( + cdi_spec_mount_path(0), + TEST_CDI_SPEC_DIR, + )], + ); + + temp_env::with_var("XDG_STATE_HOME", Some(state_dir.path()), || { + write_cdi_context_file(&sandbox, &config, &context).expect("write CDI context"); + let path = cdi_context_host_path(&sandbox, &config).expect("context path"); + let contents = fs::read(&path).expect("read CDI context"); + let parsed: CdiContext = serde_json::from_slice(&contents).expect("parse CDI context"); + assert_eq!(parsed, context); + cleanup_cdi_context_file(&sandbox, &config); + assert!(!path.exists()); + }); +} + #[test] fn build_container_create_body_omits_devices_without_resolved_default_cdi_devices() { - let mut config = runtime_config(); - config.gpu.cdi_supported = true; + let config = runtime_config(true); let mut sandbox = test_sandbox(); sandbox.spec.as_mut().unwrap().resource_requirements = Some(gpu_resources(None)); @@ -2466,8 +2576,7 @@ fn build_container_create_body_omits_devices_without_resolved_default_cdi_device #[test] fn build_container_create_body_passes_explicit_cdi_device_id_through() { - let mut config = runtime_config(); - config.gpu.cdi_supported = true; + let config = runtime_config(true); let mut sandbox = test_sandbox(); let spec = sandbox.spec.as_mut().unwrap(); spec.resource_requirements = Some(gpu_resources(None)); @@ -2490,8 +2599,7 @@ fn build_container_create_body_passes_explicit_cdi_device_id_through() { #[test] fn build_container_create_body_rejects_gpu_count_mismatched_cdi_devices() { - let mut config = runtime_config(); - config.gpu.cdi_supported = true; + let config = runtime_config(true); let mut sandbox = test_sandbox(); let spec = sandbox.spec.as_mut().unwrap(); spec.resource_requirements = Some(gpu_resources(Some(2))); @@ -2518,7 +2626,7 @@ fn build_container_create_body_rejects_cdi_devices_without_gpu_request() { .unwrap() .driver_config = Some(cdi_devices_config(&["nvidia.com/gpu=0"])); - let err = build_container_create_body(&sandbox, &runtime_config()).unwrap_err(); + let err = build_container_create_body(&sandbox, &runtime_config(false)).unwrap_err(); assert_eq!(err.code(), tonic::Code::InvalidArgument); assert!(err.message().contains("requires a gpu request")); } @@ -2530,15 +2638,14 @@ fn build_container_create_body_rejects_empty_cdi_devices() { spec.resource_requirements = Some(gpu_resources(None)); spec.template.as_mut().unwrap().driver_config = Some(cdi_devices_config(&[])); - let err = build_container_create_body(&sandbox, &runtime_config()).unwrap_err(); + let err = build_container_create_body(&sandbox, &runtime_config(false)).unwrap_err(); assert_eq!(err.code(), tonic::Code::InvalidArgument); assert!(err.message().contains("non-empty list")); } #[test] fn driver_default_gpu_selection_consumes_distinct_devices_for_creates() { - let mut config = runtime_config(); - config.gpu.cdi_supported = true; + let config = runtime_config(true); let driver = test_driver_with_config(config); driver.gpu_selector.refresh( CdiGpuInventory::new(["nvidia.com/gpu=0", "nvidia.com/gpu=1"]), @@ -2670,7 +2777,7 @@ fn require_sandbox_identifier_rejects_when_id_and_name_are_empty() { #[test] fn build_container_create_body_uses_bridge_network() { - let create_body = build_container_create_body(&test_sandbox(), &runtime_config()).unwrap(); + let create_body = build_container_create_body(&test_sandbox(), &runtime_config(false)).unwrap(); let host_config = create_body.host_config.expect("host_config is populated"); assert_eq!( @@ -2696,7 +2803,7 @@ fn build_container_create_body_uses_runtime_namespace_label() { // with that empty value would not match subsequent list/get/find // queries (which filter on `config.sandbox_namespace`), leaking // sandboxes that the driver itself cannot observe. - let mut config = runtime_config(); + let mut config = runtime_config(false); config.sandbox_namespace = "tenant-a".to_string(); let mut sandbox = test_sandbox(); sandbox.namespace = "ignored-by-driver".to_string(); diff --git a/crates/openshell-supervisor-process/src/process.rs b/crates/openshell-supervisor-process/src/process.rs index 77dd52391d..4ef3d57ae1 100644 --- a/crates/openshell-supervisor-process/src/process.rs +++ b/crates/openshell-supervisor-process/src/process.rs @@ -2294,6 +2294,9 @@ pub fn drop_privileges_with_identity( #[cfg(target_os = "linux")] if target_uid != nix::unistd::geteuid() { + // Resolve the name for initgroups only for the existing explicit-policy + // path. OCI-derived users carry a numeric UID from the bounded parser and + // must not be looked up again through NSS. let user_name_is_numeric = user_name.is_some_and(|n| n.parse::().is_ok()); let initgroups_name = if user_name.is_some() && !user_name_is_numeric && resolved_identity.uid().is_none() { diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index e46ebf8a23..cac86d6c94 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -191,6 +191,17 @@ idempotent start request. Explicitly stopped sandboxes remain stopped. For GPU-backed Docker sandboxes, configure Docker CDI before starting the gateway so OpenShell can detect the daemon capability. +For Docker GPU/CDI sandboxes, OpenShell uses Docker's selected CDI device IDs +and daemon-reported CDI spec directories to build a supervisor-only CDI +context. The driver mounts the spec directories read-only into the sandbox +container. Before creation, it writes a gateway-owned `cdi-context.json` and +bind-mounts it read-only into the supervisor. If context or token creation +fails, the driver removes the created state files; if container creation or +start fails, it also removes the container and state files. The supervisor +resolves the context inside the sandbox and derives the inner filesystem and +supplemental group requirements from CDI specs. Non-GPU Docker sandboxes do not +receive the CDI context, spec mounts, or CDI-derived policy changes. + ### Docker Driver Config Mounts Docker driver config accepts user-supplied `volume` and `tmpfs` mounts. It also