feat(driver-podman): add userns config with supervisor bind-mount fallback

Add a `userns` option to the Podman compute driver that maps to
Podman's user namespace modes.  The mode string is split on the first
colon into the API's `nsmode` and `value` fields so parameterized
values like `auto:size=65536` and `keep-id:uid=1000,gid=1000` are
forwarded correctly.  When the mode is `auto`, the container spec
also sets `idmappings.AutoUserNs = true` as required by the API.

An allowlist validates the mode at startup: `auto` and `keep-id`
accept optional parameters; `host`, `private`, and `nomap` reject
them; everything else is an error.

Podman image volumes use overlay mounts internally and the kernel
does not support idmapped mounts on overlay (`mount_setattr` returns
EINVAL).  When userns is configured (any mode except `host`), the
driver extracts the supervisor binary from the image to a host-side
cache and bind-mounts it instead of using an image volume.

Configurable via TOML `userns = "auto"`, CLI `--userns`, or
environment variable `OPENSHELL_PODMAN_USERNS`.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Author: giuseppe

.agents/skills/debug-openshell-cluster/SKILL.md

@@ -216,6 +216,24 @@ Common findings:
   cannot bypass slirp4netns host-loopback isolation. Do not work around
   discovery failures by broadening the primary gateway listener to `0.0.0.0`.
 
+When `userns` is configured (e.g. `userns = "auto"` or `userns = "keep-id"`):
+
+- Supervisor delivery uses bind-mount fallback instead of image volumes because
+  overlay mounts do not support `idmapped` mounts. The supervisor binary is
+  extracted from the supervisor image and cached at
+  `$XDG_DATA_HOME/openshell/podman-supervisor/` (typically
+  `~/.local/share/openshell/podman-supervisor/`).
+- Stale cache: if the supervisor image is updated but the cached binary is not
+  refreshed, sandbox creation may fail with an ELF validation error or version
+  mismatch. Remove the cache directory and retry.
+- `auto` mode requires subuid/subgid ranges for the current user in
+  `/etc/subuid` and `/etc/subgid`. If missing, Podman returns a user-namespace
+  mapping error at container creation.
+- `private` mode is not supported — it requires explicit UID/GID mappings that
+  the driver does not expose. Use `auto`, `keep-id`, or `no-map` instead.
+- `nomap` (without hyphen) is accepted as input but canonicalized to `no-map`
+  for Podman's API.
+
 ### Step 6: Check Kubernetes Helm Gateways
 
 ```bash

architecture/compute-runtimes.md

@@ -117,7 +117,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
 | Runtime | Best fit | Sandbox boundary | Notes |
 |---|---|---|---|
 | Docker | Local development with Docker available. | Container plus nested sandbox namespace. | Uses host networking so loopback gateway endpoints work from the supervisor. |
-| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API, OCI image volumes, and CDI GPU devices when available. |
+| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API and CDI GPU devices when available. Delivers the supervisor via OCI image volume by default; falls back to extracting the binary to a host-side cache and bind-mounting it when `userns` is configured (overlay does not support idmapped mounts). |
 | Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
 | VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
 | Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
@@ -170,7 +170,7 @@ The supervisor must be available inside each sandbox workload:
 | Runtime | Delivery model |
 |---|---|
 | Docker | Bind-mounted local supervisor binary, or a binary extracted from the configured supervisor image. |
-| Podman | Read-only OCI image volume containing the supervisor binary. |
+| Podman | Read-only OCI image volume by default; host-cached bind mount when `userns` is configured. |
 | Kubernetes | Supervisor image side-loaded into the sandbox pod by image volume or init container. |
 | VM | Embedded in the guest rootfs bundle. |
 | Extension | Defined by the out-of-tree driver. |

crates/openshell-driver-podman/README.md

@@ -360,6 +360,7 @@ Podman resources after out-of-band container removal or label drift.
 | `OPENSHELL_SANDBOX_PROXY_AUTH_FILE` | `--sandbox-proxy-auth-file` | unset | Path to a file containing the proxy credentials as `user:pass`. Staged as a root-only Podman secret so credentials never appear in config or container metadata. Requires the insecure-auth acknowledgement below. |
 | `OPENSHELL_SANDBOX_PROXY_AUTH_ALLOW_INSECURE` | `--sandbox-proxy-auth-allow-insecure` | unset | Explicit acknowledgement (`true`) that the credential is sent as cleartext Basic auth over the plain-TCP connection to the `http://` proxy. Required when the auth file is set; rejected when it is not. |
 | `OPENSHELL_SANDBOX_PROXY_CONNECT_BY_HOSTNAME` | `--sandbox-proxy-connect-by-hostname` | unset | Send the destination hostname in CONNECT requests instead of a validated IP. Last resort for proxies whose ACLs filter on hostnames: the proxy then resolves the name itself, so sandbox SSRF/`allowed_ips` validation no longer binds the connection. |
+| `OPENSHELL_PODMAN_USERNS` | `--userns` | unset | User namespace mode for sandbox containers (e.g. `auto`). When unset, containers use the default user namespace. |
 
 Through the gateway, the same settings are the `https_proxy`, `no_proxy`,
 `proxy_auth_file`, `proxy_auth_allow_insecure`, and

crates/openshell-driver-podman/src/client.rs

@@ -513,6 +513,32 @@ impl PodmanClient {
         }
     }
 
+    /// Download a file from a container as a tar archive.
+    ///
+    /// Calls `GET /libpod/containers/{name}/archive?path={path}` and returns
+    /// the raw tar bytes. The container does not need to be running.
+    pub async fn copy_from_container(
+        &self,
+        name: &str,
+        path: &str,
+    ) -> Result<Bytes, PodmanApiError> {
+        validate_name(name)?;
+        let encoded_path = url_encode(path);
+        let (status, bytes) = self
+            .request(
+                hyper::Method::GET,
+                &format!("/libpod/containers/{name}/archive?path={encoded_path}"),
+                None,
+                API_TIMEOUT,
+            )
+            .await?;
+        if status.is_success() {
+            Ok(bytes)
+        } else {
+            Err(error_from_response(status.as_u16(), &bytes))
+        }
+    }
+
     /// Inspect a container by name or ID.
     pub async fn inspect_container(&self, name: &str) -> Result<ContainerInspect, PodmanApiError> {
         validate_name(name)?;

crates/openshell-driver-podman/src/config.rs

@@ -185,6 +185,9 @@ pub struct PodmanComputeConfig {
     /// pointing the gateway host at the corporate resolver so validated-IP
     /// CONNECT works in split-horizon networks.
     pub proxy_connect_by_hostname: Option<bool>,
+    /// User namespace mode for sandbox containers (e.g. `auto`).
+    /// When unset, containers use the default user namespace.
+    pub userns: Option<String>,
 }
 
 pub const DEFAULT_HEALTH_CHECK_INTERVAL_SECS: u64 = 10;
@@ -328,6 +331,53 @@ impl PodmanComputeConfig {
         Ok(())
     }
 
+    /// Validate and canonicalize the optional `userns` mode.
+    ///
+    /// Supported modes: `auto` (with optional params, e.g. `auto:size=65536`),
+    /// `host`, `keep-id` (with optional params), and `no-map` (alias `nomap`).
+    /// Modes that don't accept parameters (`host`, `no-map`) are rejected when
+    /// a colon-separated suffix is present.
+    ///
+    /// On success, `self.userns` is rewritten with the canonical lowercase
+    /// mode string so downstream code can rely on exact matches.
+    pub fn canonicalize_userns(&mut self) -> Result<(), crate::client::PodmanApiError> {
+        let Some(mode) = self.userns.as_deref() else {
+            return Ok(());
+        };
+        let (base, has_params) = mode
+            .split_once(':')
+            .map_or((mode, false), |(b, _)| (b, true));
+        let canonical = match base.to_ascii_lowercase().as_str() {
+            "auto" => "auto",
+            "host" => "host",
+            "keep-id" => "keep-id",
+            "nomap" | "no-map" => "no-map",
+            _ => {
+                return Err(crate::client::PodmanApiError::InvalidInput(format!(
+                    "unsupported userns mode '{mode}'; \
+                     supported modes: auto, host, keep-id, no-map",
+                )));
+            }
+        };
+        if has_params {
+            match canonical {
+                "auto" | "keep-id" => {}
+                _ => {
+                    return Err(crate::client::PodmanApiError::InvalidInput(format!(
+                        "userns mode '{canonical}' does not accept parameters",
+                    )));
+                }
+            }
+        }
+        self.userns = Some(if has_params {
+            let params = mode.split_once(':').unwrap().1;
+            format!("{canonical}:{params}")
+        } else {
+            canonical.to_string()
+        });
+        Ok(())
+    }
+
     /// Validate optional host gateway override.
     pub fn validate_host_gateway_ip(&self) -> Result<(), crate::client::PodmanApiError> {
         let trimmed = self.host_gateway_ip.trim();
@@ -380,6 +430,7 @@ impl Default for PodmanComputeConfig {
             proxy_auth_file: None,
             proxy_auth_allow_insecure: None,
             proxy_connect_by_hostname: None,
+            userns: None,
         }
     }
 }
@@ -412,6 +463,7 @@ impl std::fmt::Debug for PodmanComputeConfig {
             .field("proxy_auth_file", &self.proxy_auth_file.is_some())
             .field("proxy_auth_allow_insecure", &self.proxy_auth_allow_insecure)
             .field("proxy_connect_by_hostname", &self.proxy_connect_by_hostname)
+            .field("userns", &self.userns)
             .finish()
     }
 }
@@ -794,4 +846,89 @@ mod tests {
         assert!(!msg.contains("OPENSHELL_PODMAN_TLS_CERT"), "{msg}");
         assert!(!msg.contains("OPENSHELL_PODMAN_TLS_KEY"), "{msg}");
     }
+
+    #[test]
+    fn canonicalize_userns_accepts_supported_modes() {
+        for mode in [
+            "auto",
+            "host",
+            "keep-id",
+            "no-map",
+            "auto:size=65536",
+            "keep-id:uid=1000,gid=1000",
+        ] {
+            let mut cfg = PodmanComputeConfig {
+                userns: Some(mode.to_string()),
+                ..PodmanComputeConfig::default()
+            };
+            cfg.canonicalize_userns()
+                .unwrap_or_else(|_| panic!("mode '{mode}' should be accepted"));
+        }
+    }
+
+    #[test]
+    fn canonicalize_userns_normalizes_case_and_aliases() {
+        let cases = [
+            ("Auto", "auto"),
+            ("HOST", "host"),
+            ("KEEP-ID:uid=1000", "keep-id:uid=1000"),
+            ("nomap", "no-map"),
+            ("no-map", "no-map"),
+            ("auto:SIZE=65536", "auto:SIZE=65536"),
+        ];
+        for (input, expected) in cases {
+            let mut cfg = PodmanComputeConfig {
+                userns: Some(input.to_string()),
+                ..PodmanComputeConfig::default()
+            };
+            cfg.canonicalize_userns()
+                .unwrap_or_else(|_| panic!("mode '{input}' should be accepted"));
+            assert_eq!(
+                cfg.userns.as_deref(),
+                Some(expected),
+                "input '{input}' should canonicalize to '{expected}'"
+            );
+        }
+    }
+
+    #[test]
+    fn canonicalize_userns_rejects_unsupported_modes() {
+        for mode in [
+            "container:foo",
+            "ns:/proc/1/ns/user",
+            "4000:5000",
+            "private",
+        ] {
+            let mut cfg = PodmanComputeConfig {
+                userns: Some(mode.to_string()),
+                ..PodmanComputeConfig::default()
+            };
+            let err = cfg
+                .canonicalize_userns()
+                .expect_err(&format!("mode '{mode}' should be rejected"));
+            let msg = err.to_string();
+            assert!(msg.contains("unsupported userns mode"), "{msg}");
+        }
+    }
+
+    #[test]
+    fn canonicalize_userns_rejects_params_on_non_parameterizable_modes() {
+        for mode in ["host:foo", "no-map:x=1"] {
+            let mut cfg = PodmanComputeConfig {
+                userns: Some(mode.to_string()),
+                ..PodmanComputeConfig::default()
+            };
+            let err = cfg
+                .canonicalize_userns()
+                .expect_err(&format!("mode '{mode}' should be rejected"));
+            let msg = err.to_string();
+            assert!(msg.contains("does not accept parameters"), "{msg}");
+        }
+    }
+
+    #[test]
+    fn canonicalize_userns_accepts_none() {
+        let mut cfg = PodmanComputeConfig::default();
+        cfg.canonicalize_userns().expect("None should be accepted");
+    }
 }