Add microVM isolation mode for project labs via Kata Containers

docs/lab-runtime-hardening.md

@@ -21,6 +21,32 @@ The backend command builder applies these Docker settings by default:
 
 The default Linux user is `root` because the upstream `kalilinux/kali-rolling` image does not ship a `kali` account. The container still runs with dropped Linux capabilities, `no-new-privileges`, no Docker socket, and no host path mounts by default. Custom lab images can provide and select an unprivileged user when stronger in-container user separation is needed.
 
+## MicroVM Isolation (Kata Containers)
+
+The default `container` isolation mode above shares the host kernel with every lab: namespaces, cgroups, and dropped capabilities narrow the attack surface, but a container escape still lands on the host kernel. Lab boundary options include a `microvm` isolation mode that runs the same lab container under a [Kata Containers](https://katacontainers.io/) OCI runtime class, so each lab gets its own guest kernel inside a hardware-virtualized microVM (Firecracker, QEMU, or Cloud Hypervisor). This is additive to, not a replacement for, the capability/network/resource hardening described above.
+
+### Enabling it
+
+- Per-lab: pass `isolation: "microvm"` (and optionally `microvmRuntimeClass`) to the lab create/start/restart API. The choice is persisted in lab metadata, so subsequent starts reuse it.
+- Host-wide default: set `PROJECT_LAB_ISOLATION_MODE=microvm`, and optionally `PROJECT_LAB_MICROVM_RUNTIME_CLASS` to override the default runtime class.
+- Runtime class names map directly to `docker run --runtime <class>` and must already be registered in the Docker daemon's `runtimes` config (`/etc/docker/daemon.json`) by a Kata Containers install:
+  - `kata-fc` (default) — Firecracker VMM. Smallest device model and strongest isolation, but only virtio net/block/vsock devices are available to the guest.
+  - `kata-qemu` — QEMU VMM. Broader device/hardware compatibility if the Kali workspace image needs devices Firecracker does not emulate.
+  - `kata-clh` — Cloud Hypervisor VMM. A middle ground between the two.
+
+### Fail-closed behavior
+
+Because isolation strength is a security property, the runtime never silently downgrades from `microvm` to plain containers. Before starting a lab with `isolation: "microvm"`, it runs `docker info --format '{{json .Runtimes}}'` and refuses to start (throwing `MicrovmRuntimeUnavailableError`) if the requested runtime class is not registered with the Docker daemon. In `dry-run` mode the availability check is included in the planned command list but is never executed or enforced.
+
+### Host setup
+
+MicroVM isolation requires a Docker host with:
+
+- Nested virtualization / bare-metal KVM access (`/dev/kvm` present, `vmx`/`svm` CPU flag set).
+- [Kata Containers](https://github.com/kata-containers/kata-containers/blob/main/docs/install/README.md) installed and registered with the Docker daemon under the runtime class names above.
+
+This is a host/infrastructure prerequisite outside of ExploitHunter.app's process; most local developer machines, CI runners, and constrained sandboxes do not expose `/dev/kvm` and cannot run `microvm` isolation at all. Use `container` isolation (the default) in those environments, and reserve `microvm` for hosts that are provisioned for it.
+
 ## Network Profiles
 
 Human profiles:
@@ -90,3 +116,4 @@ Network profile changes are bound to durable approvals. The `agentLabCommandTool
 - DNS resolution for approved targets uses the container's configured DNS resolver. The iptables rules match against resolved IP addresses at connection time, not domain names. This means the hostname-based allowlisting is resolved at the time each connection is made.
 - The iptables enforcement runs inside the container and can theoretically be modified by a root user inside the container. The `--cap-drop ALL` before adding `NET_ADMIN`/`NET_RAW` means only these specific networking capabilities are available.
 - For stronger enforcement, pair with a dedicated Docker network that has its own egress firewall, or use a sidecar proxy (Envoy, SOCKS5) in the container network namespace.
+- Namespace/cgroup isolation shares the host kernel with the lab workload. For labs where a kernel-level container escape is an unacceptable risk (for example, running untrusted exploit code or malware samples), use `microvm` isolation (see above) so the lab runs in its own guest kernel instead of the host's.

src/mastra/config/storage.ts

@@ -1,3 +1,6 @@
+import { mkdirSync } from "node:fs";
+import { dirname } from "node:path";
+
 import { FilesystemStore, InMemoryStore, MastraCompositeStore } from "@mastra/core/storage";
 import nextEnv from "@next/env";
 import { ClickhouseStoreVNext } from "@mastra/clickhouse";
@@ -33,9 +36,21 @@ function localPath(uri: string, scheme: "duckdb" | "lance" | "lancedb") {
   return uri.slice(`${scheme}://`.length) || ".";
 }
 
+// libSQL's local sqlite backend fails to open a file whose parent directory does not
+// exist yet (e.g. a fresh checkout before `.data/` has ever been created); `libsql://`
+// remote URLs and `:memory:` are not local files and need no directory.
+function ensureLocalSqliteDirectory(uri: string) {
+  if (!uri.startsWith("file:")) return;
+  const path = uri.slice("file:".length);
+  if (!path || path === ":memory:") return;
+  const dir = dirname(path);
+  if (dir && dir !== ".") mkdirSync(dir, { recursive: true });
+}
+
 async function createMastraStorageAdapter(config: MastraStorageConfig, id: string): Promise<MastraCompositeStore> {
   switch (config.adapter) {
     case "libsql":
+      ensureLocalSqliteDirectory(config.uri);
       return new LibSQLStore({ id, url: config.uri, connectionTimeoutMs: Number.parseInt(process.env.SQLITE_BUSY_TIMEOUT_MS ?? "5000", 10) });
     case "postgres": return new PostgresStore({ id, connectionString: config.uri });
     case "redis": return new RedisStore({ id, connectionString: config.uri });

src/server/labs/docker-plan.ts

@@ -11,7 +11,10 @@ import {
   LAB_HARDENING_LABELS,
   LAB_HOME_PATH,
   LAB_WORKSPACE_PATH,
+  SAFE_MICROVM_RUNTIME_CLASS,
   dockerHardeningArgs,
+  resolveLabIsolationMode,
+  resolveMicrovmRuntimeClass,
 } from "./hardening";
 import { dockerNetworkArgs, getLabNetworkProfile, buildEgressEnforcementArgs, buildEgressIptablesScript } from "./network-profiles";
 import type { DockerCommandSpec, LabContainerIdentity, LabContainerOptions, LabResourceLimits } from "./types";
@@ -137,9 +140,14 @@ export const buildRunLabCommand = (options: LabContainerOptions): DockerCommandS
   const user = options.user ?? DEFAULT_KALI_LAB_USER;
   const networkProfile = getLabNetworkProfile(options.boundary, options.networkProfile);
   const limits = mergeResourceLimits(options);
+  const isolation = resolveLabIsolationMode(options.isolation);
+  const microvmRuntimeClass = isolation === "microvm" ? resolveMicrovmRuntimeClass(options.microvmRuntimeClass) : null;
 
   assertSafeValue("image ref", image, SAFE_IMAGE_REF);
   assertSafeValue("user", user, SAFE_USER);
+  if (microvmRuntimeClass) {
+    assertSafeValue("microVM runtime class", microvmRuntimeClass, SAFE_MICROVM_RUNTIME_CLASS);
+  }
 
   return {
     command: "docker",
@@ -154,10 +162,13 @@ export const buildRunLabCommand = (options: LabContainerOptions): DockerCommandS
       user,
       "--workdir",
       LAB_WORKSPACE_PATH,
+      ...(microvmRuntimeClass ? ["--runtime", microvmRuntimeClass] : []),
       ...labelArgs({
         ...identity.labels,
         "exploit-hunter.network-profile": networkProfile.id,
         "exploit-hunter.image-ref": image,
+        "exploit-hunter.isolation": isolation,
+        ...(microvmRuntimeClass ? { "exploit-hunter.microvm-runtime-class": microvmRuntimeClass } : {}),
       }),
       ...dockerHardeningArgs(options.boundary, limits),
       ...buildEgressEnforcementArgs({
@@ -202,6 +213,19 @@ export const buildStartLabCommands = (options: LabContainerOptions): DockerComma
   buildRunLabCommand(options),
 ];
 
+export const buildMicrovmRuntimeCheckCommand = (options: LabContainerOptions): DockerCommandSpec | null => {
+  if (resolveLabIsolationMode(options.isolation) !== "microvm") {
+    return null;
+  }
+  const runtimeClass = resolveMicrovmRuntimeClass(options.microvmRuntimeClass);
+  assertSafeValue("microVM runtime class", runtimeClass, SAFE_MICROVM_RUNTIME_CLASS);
+  return {
+    command: "docker",
+    args: ["info", "--format", "{{json .Runtimes}}"],
+    reason: `Verify the ${runtimeClass} microVM runtime is registered with the Docker daemon before starting an isolated lab.`,
+  };
+};
+
 export const buildStopLabCommand = (options: LabContainerOptions): DockerCommandSpec => ({
   command: "docker",
   args: ["stop", labContainerIdentity(options).containerName],

src/server/labs/hardening.ts

@@ -1,4 +1,4 @@
-import type { LabBoundary, LabResourceLimits } from "./types";
+import type { LabBoundary, LabIsolationMode, LabResourceLimits } from "./types";
 
 export const PREVIOUS_KALI_LAB_IMAGE = "exploit-hunter/kali-workspace:latest";
 export const DEFAULT_KALI_LAB_IMAGE = "exploit-hunter/kali-workspace:iproute2";
@@ -68,3 +68,23 @@ export const dockerHardeningArgs = (boundary: LabBoundary, limits: LabResourceLi
   "--label",
   `exploit-hunter.lab.boundary=${boundary}`,
 ];
+
+// Kata Containers registers hardware-virtualized OCI runtime classes with the Docker
+// daemon (`/etc/docker/daemon.json` "runtimes"); each name below selects a different VMM.
+// kata-fc (Firecracker) is the strictest microVM boundary but supports only virtio
+// net/block/vsock devices; kata-qemu and kata-clh (Cloud Hypervisor) trade some of that
+// isolation minimalism for broader device/hardware compatibility with the Kali image.
+export const MICROVM_RUNTIME_CLASSES = ["kata-fc", "kata-qemu", "kata-clh"] as const;
+export const DEFAULT_LAB_ISOLATION_MODE: LabIsolationMode = "container";
+export const DEFAULT_MICROVM_RUNTIME_CLASS: (typeof MICROVM_RUNTIME_CLASSES)[number] = "kata-fc";
+export const SAFE_MICROVM_RUNTIME_CLASS = /^[a-zA-Z0-9][a-zA-Z0-9._-]{0,63}$/;
+
+export const resolveLabIsolationMode = (isolation?: LabIsolationMode): LabIsolationMode => {
+  if (isolation === "microvm" || isolation === "container") {
+    return isolation;
+  }
+  return process.env.PROJECT_LAB_ISOLATION_MODE === "microvm" ? "microvm" : DEFAULT_LAB_ISOLATION_MODE;
+};
+
+export const resolveMicrovmRuntimeClass = (runtimeClass?: string): string =>
+  runtimeClass?.trim() || process.env.PROJECT_LAB_MICROVM_RUNTIME_CLASS?.trim() || DEFAULT_MICROVM_RUNTIME_CLASS;

src/server/labs/runtime.ts

@@ -8,13 +8,14 @@ import {
   buildBuildLabImageCommand,
   buildDestroyLabContainerCommand,
   buildInspectLabImageCommand,
+  buildMicrovmRuntimeCheckCommand,
   buildProvisionLabCommands,
   buildRunLabCommand,
   isManagedKaliLabImage,
   buildStopLabCommand,
   labContainerIdentity,
 } from "./docker-plan";
-import { DEFAULT_KALI_LAB_IMAGE } from "./hardening";
+import { DEFAULT_KALI_LAB_IMAGE, resolveMicrovmRuntimeClass } from "./hardening";
 import { buildEgressIptablesScript, buildRuntimeFirewallScript } from "./network-profiles";
 import type {
   DockerCommandSpec,
@@ -31,6 +32,17 @@ import type {
 
 const execFileAsync = promisify(execFile);
 
+export class MicrovmRuntimeUnavailableError extends Error {
+  constructor(runtimeClass: string, cause?: string) {
+    super(
+      `MicroVM isolation was requested but the "${runtimeClass}" Docker runtime is not registered on this host` +
+        `${cause ? ` (${cause})` : ""}. Install and register Kata Containers (see docs/lab-runtime-hardening.md) ` +
+        `or set PROJECT_LAB_ISOLATION_MODE=container.`,
+    );
+    this.name = "MicrovmRuntimeUnavailableError";
+  }
+}
+
 export interface DockerCommandRunner {
   isAvailable(): Promise<boolean>;
   run(
@@ -98,13 +110,18 @@ export class ProjectLabRuntime {
 
   async start(options: LabContainerOptions): Promise<LabRuntimeResult> {
     const mode = await this.resolveMode();
+    const microvmCheckCommand = buildMicrovmRuntimeCheckCommand(options);
+    if (microvmCheckCommand && mode === "docker") {
+      await this.assertMicrovmRuntimeAvailable(options, microvmCheckCommand);
+    }
     const imageCommands = await this.ensureManagedImage(options, mode);
     const provisionCommands = buildProvisionLabCommands(options);
     const commands = [...imageCommands, ...provisionCommands];
     const identity = labContainerIdentity(options);
 
     if (mode === "dry-run") {
       const planned = [
+        ...(microvmCheckCommand ? [microvmCheckCommand] : []),
         ...commands,
         {
           command: "docker" as const,
@@ -392,6 +409,31 @@ export class ProjectLabRuntime {
     }
   }
 
+  private async assertMicrovmRuntimeAvailable(options: LabContainerOptions, checkCommand: DockerCommandSpec): Promise<void> {
+    const runtimeClass = resolveMicrovmRuntimeClass(options.microvmRuntimeClass);
+    let stdout: string;
+    try {
+      stdout = (await this.runner.run(checkCommand, { timeoutMs: 5_000 })).stdout;
+    } catch (error) {
+      throw new MicrovmRuntimeUnavailableError(runtimeClass, error instanceof Error ? error.message : String(error));
+    }
+
+    let runtimes: unknown;
+    try {
+      runtimes = JSON.parse(stdout || "{}");
+    } catch {
+      runtimes = {};
+    }
+
+    const isRegistered =
+      runtimes !== null && typeof runtimes === "object" && !Array.isArray(runtimes) &&
+      Object.prototype.hasOwnProperty.call(runtimes, runtimeClass);
+
+    if (!isRegistered) {
+      throw new MicrovmRuntimeUnavailableError(runtimeClass);
+    }
+  }
+
   private async applyEgressIptables(options: LabContainerOptions, identity: ReturnType<typeof labContainerIdentity>): Promise<void> {
     const script = buildEgressIptablesScript({
       profileId: options.networkProfile ?? "offline",

src/server/labs/service.ts

@@ -6,13 +6,19 @@ import type {
   LabCredentialMount,
   HumanLabNetworkProfileId,
   LabFirewallConfigInput,
+  LabIsolationMode,
   LabRuntimeResult,
   NormalizedLabCreateInput,
   ProjectLabRepository,
   ProjectLabStatusView,
 } from "./types";
 import { ProjectLabRuntime } from "./runtime";
-import { DEFAULT_KALI_LAB_IMAGE, PREVIOUS_KALI_LAB_IMAGE, UPSTREAM_KALI_LAB_IMAGE } from "./hardening";
+import {
+  DEFAULT_KALI_LAB_IMAGE,
+  PREVIOUS_KALI_LAB_IMAGE,
+  UPSTREAM_KALI_LAB_IMAGE,
+  resolveLabIsolationMode,
+} from "./hardening";
 import { buildUfwCommandPlan, normalizeLabFirewallConfig } from "./network-profiles";
 
 const LEGACY_SIMULATED_IMAGE_REF = "exploit-hunter-lab:simulated";
@@ -100,11 +106,15 @@ export class ProjectLabService {
     }
 
     const networkProfile = readHumanNetworkProfile(input.networkProfile ?? lab.metadata.networkProfile);
+    const isolation = readLabIsolationMode(input.isolation ?? lab.metadata.isolation);
+    const microvmRuntimeClass = readMicrovmRuntimeClass(input.microvmRuntimeClass ?? lab.metadata.microvmRuntimeClass);
     const runtimeResult = await this.runtime.start({
       projectId,
       boundary: "human",
       image: lab.image_ref,
       networkProfile,
+      isolation,
+      microvmRuntimeClass,
       credentialMounts: readCredentialMounts(input.credentialMounts ?? lab.metadata.credentialMounts),
     });
     const runtimeId = lab.runtime_id ?? runtimeResult.container.containerName;
@@ -119,7 +129,7 @@ export class ProjectLabService {
       started_at: lab.started_at ?? timestamp(),
       stopped_at: null,
       destroyed_at: null,
-      metadata: mergeMetadata(lab.metadata, labPreferenceMetadata({ ...input, networkProfile })),
+      metadata: mergeMetadata(lab.metadata, labPreferenceMetadata({ ...input, networkProfile, isolation, microvmRuntimeClass })),
     });
 
     return this.statusFromLab(projectId, running);
@@ -167,11 +177,15 @@ export class ProjectLabService {
       networkProfile: "offline",
     });
     const networkProfile = readHumanNetworkProfile(input.networkProfile ?? lab.metadata.networkProfile);
+    const isolation = readLabIsolationMode(input.isolation ?? lab.metadata.isolation);
+    const microvmRuntimeClass = readMicrovmRuntimeClass(input.microvmRuntimeClass ?? lab.metadata.microvmRuntimeClass);
     const startResult = await this.runtime.start({
       projectId,
       boundary: "human",
       image: lab.image_ref,
       networkProfile,
+      isolation,
+      microvmRuntimeClass,
       credentialMounts: readCredentialMounts(input.credentialMounts ?? lab.metadata.credentialMounts),
     });
     const runtimeResult = mergeRuntimeResults(destroyResult, startResult);
@@ -186,7 +200,7 @@ export class ProjectLabService {
       started_at: timestamp(),
       stopped_at: null,
       destroyed_at: null,
-      metadata: mergeMetadata(lab.metadata, labPreferenceMetadata({ ...input, networkProfile })),
+      metadata: mergeMetadata(lab.metadata, labPreferenceMetadata({ ...input, networkProfile, isolation, microvmRuntimeClass })),
     });
 
     return this.statusFromLab(projectId, running);
@@ -316,9 +330,14 @@ function labDefaults(input: LabCreateInput): NormalizedLabCreateInput {
 }
 
 function labPreferenceMetadata(input: LabCreateInput): JsonObject {
+  const microvmRuntimeClass = readMicrovmRuntimeClass(input.microvmRuntimeClass);
   return mergeMetadata(
     mergeMetadata(sanitizeJsonObject(input.metadata), credentialMountMetadata(input.credentialMounts)),
-    { networkProfile: readHumanNetworkProfile(input.networkProfile) },
+    {
+      networkProfile: readHumanNetworkProfile(input.networkProfile),
+      isolation: readLabIsolationMode(input.isolation),
+      ...(microvmRuntimeClass ? { microvmRuntimeClass } : {}),
+    },
   );
 }
 
@@ -328,6 +347,14 @@ function readHumanNetworkProfile(value: unknown): HumanLabNetworkProfileId {
     : DEFAULT_HUMAN_NETWORK_PROFILE;
 }
 
+function readLabIsolationMode(value: unknown): LabIsolationMode {
+  return resolveLabIsolationMode(value === "microvm" || value === "container" ? value : undefined);
+}
+
+function readMicrovmRuntimeClass(value: unknown): string | undefined {
+  return typeof value === "string" && value.trim() ? value.trim() : undefined;
+}
+
 function readFirewallConfig(value: unknown): LabFirewallConfigInput | undefined {
   return value && typeof value === "object" && !Array.isArray(value)
     ? value as LabFirewallConfigInput

src/server/labs/types.ts

@@ -33,6 +33,8 @@ export interface LabCreateInput {
   networkProfile?: HumanLabNetworkProfileId;
   firewall?: LabFirewallConfigInput;
   credentialMounts?: LabCredentialMount[];
+  isolation?: LabIsolationMode;
+  microvmRuntimeClass?: string;
   metadata?: JsonObject;
 }
 
@@ -62,6 +64,13 @@ export interface ProjectLabRepository {
 
 export type LabBoundary = "human" | "agent";
 
+/**
+ * "container" is the default Linux-namespace/cgroup boundary (see hardening.ts).
+ * "microvm" runs the lab under a hardware-virtualized Kata Containers runtime class
+ * for kernel-level isolation between the lab workload and the Docker host.
+ */
+export type LabIsolationMode = "container" | "microvm";
+
 export type HumanLabNetworkProfileId = "offline" | "package-egress" | "full";
 
 export type AgentLabNetworkProfileId = "offline" | "approved-targets" | "package-egress" | "full";
@@ -107,6 +116,8 @@ export interface LabContainerOptions extends LabProjectRef {
   networkName?: string;
   approvedTargets?: string[];
   credentialMounts?: LabCredentialMount[];
+  isolation?: LabIsolationMode;
+  microvmRuntimeClass?: string;
 }
 
 export interface DockerCommandSpec {