Running builah on unprivileged gitlab kubernetes executor

[shahGnp]

$ buildah images
Error during unshare(CLONE_NEWUSER): Operation not permitted
time="2025-09-04T11:48:34Z" level=error msg="parsing PID \"\": strconv.Atoi: parsing \"\": invalid syntax"

I'm seeing this issue with buildah when it's run on gitlab kubernetes executor in unprivileged mode.

Helm values

gitlabUrl: https://gitlab.com
runnerRegistrationToken: xxx-my-token-xxx
unregisterRunners: true

## For RBAC support:
rbac:
  create: true
  rules:
    - resources: ["events"]
      verbs: ["list", "watch"]
    - resources: ["namespaces"]
      verbs: ["create", "delete"]
    - resources: ["pods"]
      verbs: ["create","delete","get","list","watch"]
    - apiGroups: [""]
      resources: ["pods/attach","pods/exec"]
      verbs: ["get","create","patch","delete"]
    - apiGroups: [""]
      resources: ["pods/log"]
      verbs: ["get","list"]
    - resources: ["secrets"]
      verbs: ["create","delete","get","update"]
    - resources: ["serviceaccounts"]
      verbs: ["get"]
    - resources: ["services"]
      verbs: ["create","get"]

  clusterWideAccess: false

  ## Use podSecurity Policy
  ## ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/
  podSecurityPolicy:
    enabled: true
    resourceNames:
      - gitlab-runner

securityContext:
  capabilities:
    drop: ["ALL"]
  allowPrivilegeEscalation: false
  runAsNonRoot: true
  seccompProfile:
    type: "RuntimeDefault"

serviceAccount:
  create: true
  name: "gitlab-runner"

runners:
  config: |
    [[runners]]
      environment = [
        "HOME=/ephemeral_build_home",
        "TMPDIR=/ephemeral_tmp_dir"
      ]
      [runners.feature_flags]
        FF_USE_ADVANCED_POD_SPEC_CONFIGURATION = true
      [runners.kubernetes]
        [[runners.kubernetes.volumes.empty_dir]]
          name = "build-home"
          mount_path = "/ephemeral_build_home"
          # medium = "Memory"
        [[runners.kubernetes.volumes.empty_dir]]
          name = "tmp-dir"
          mount_path = "/ephemeral_tmp_dir"
          # medium = "Memory"
        [[runners.kubernetes.pod_spec]]
          name = "securityContexts"
          patch = '''
            securityContext:
              runAsUser: 1000
              runAsGroup: 1000
              fsGroup: 1000
              runAsNonRoot: true
              seccompProfile:
                type: "RuntimeDefault"
            # set securityContext for individual containers
            containers:
            - name: build
              securityContext:
                readOnlyRootFilesystem: true
                capabilities:
                  drop: ["ALL"]
                allowPrivilegeEscalation: false
            - name: helper
              securityContext:
                readOnlyRootFilesystem: true
                capabilities:
                  drop: ["ALL"]
                allowPrivilegeEscalation: false
            initContainers:
            - name: init-permissions
              securityContext:
                readOnlyRootFilesystem: true
                capabilities:
                  drop: ["ALL"]
                allowPrivilegeEscalation: false
          '''
          patch_type = "strategic"

[nalind]

buildah needs to be able to operate as root with CAP_SYS_ADMIN, whether it's started that way in a privileged container, or with enough privileges to allow it to set up a user namespace in which it can do these things.

The latter requires that unshare(2) is available, and that CAP_SETFCAP (if started as UID 0, which I think can be in a hostUsers: false container) or CAP_SETUID/CAP_SETGID (if started as another UID, with entries in the /etc/subuid and /etc/subgid in the container that allow the newuidmap and newgidmap in the container to set up ID mappings) are available. There might be other requirements I'm forgetting at the moment.

[shahGnp]

Thank you for you response. I also realized that it needs CAP_SYS_ADMIN ability which is a higher privilege. However, I also tried removing all the restrictions and I faced this issue.

$ buildah images
REPOSITORY   TAG   IMAGE ID   CREATED   SIZE
$ buildah bud --format "$BUILDAH_FORMAT" --storage-driver "$STORAGE_DRIVER" --isolation=chroot -t "$FQ_IMAGE_NAME" .
STEP 1/11: FROM python:3.9.2
Resolved "python" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/python:3.9.2...
Getting image source signatures
Copying blob sha256:004f1eed87df3f75f5e2a1a649fa7edd7f713d1300532fd0909bb39cd48437d7
Copying blob sha256:5d6f1e8117dbb1c6a57603cb4f321a861a08105a81bcc6b01b0ec2b78c8523a5
Copying blob sha256:48c2faf66abec3dce9f54d6722ff592fce6dd4fb58a0d0b72282936c6598a3b3
Copying blob sha256:234b70d0479d7f16d7ee8d04e4ffdacc57d7d14313faf59d332f18b2e9418743
Copying blob sha256:6fa07a00e2f029c4b2c7f177a2b696f1b3510040cde4f5bb06ddbca98e7fbf76
Copying blob sha256:04a31b4508b8e95fb3cb25486c4068185054895b12e0611e386a002ee9c0e07c
Copying blob sha256:1823d93d969893b61303752fa11b710c4befae6f15a775fad2c2c0bcb619cf2e
Copying blob sha256:2eee1b8e253830859988454f07e1e98387cf5e3730e7eedee333aa7fd1a04be2
Copying blob sha256:b48672c0140e5d73a8b044e4c1c229f3cc88fd8ff645c7fc6f774c830ae1f8d4
time="2025-09-04T12:32:06Z" level=error msg="While applying layer: ApplyLayer stdout:  stderr: remount /, flags: 0x44000: permission denied exit status 1"
Error: creating build container: unable to copy from source docker://python:3.9.2: copying system image from manifest list: writing blob: adding layer with blob "sha256:004f1eed87df3f75f5e2a1a649fa7edd7f713d1300532fd0909bb39cd48437d7"/""/"sha256:4ef4adca5c3b96107661045d947bf51f1f27f47c2557c9eea367c32a8bc8c399": ApplyLayer stdout:  stderr: remount /, flags: 0x44000: permission denied exit status 1
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: command terminated with exit code 1

Why is it saying permission denied, I have not restrcited it in anyway.

[nalind]

The error's occurring in a child process that's attempting to extract the contents of a layer in a chroot-like manner to avoid possible issues if the layer includes items that, through weird pathname shenanigans, might otherwise cause it to write (or overwrite) content outside of the directory where the layer's contents should go.

At the point where the error occurs, it's created a new mount namespace, and is attempting to use mount() to reset the propagation for all of the mountpoints in the new mount namespace to "private", and that's failing, and I'm not sure what, other than lack of CAP_SYS_ADMIN, would cause that. Maybe readOnlyRootFilesystem: true, but I'm not certain of that.

Can you re-run with --debug? The first thing it'll do in that case is log the effective capabilities it has, and if it creates a new user namespace for itself, the copy of itself that it runs in that new user namespace will do the same. That'll let us rule out missing capabilities, at least.