Add cgroupv2 support to Docker CPI

[mike-jc]

Currently, Docker CPI supports only Control Groups v1, which makes it unusable on Ubuntu Noble, where the Bosh team has decided to use Control Groups v2.

Is it possible to add cgroupv2 support to Docker CPI?

[a-hassanin]

Relevant for Noble: cloudfoundry/community#892

[beyhan]

We would love to review a pr for this.

[maxmoehl]

@beyhan and I spoke a about this offline, I will first give a bit more context on why we have raised this issue:

As part of the haproxy-boshrelease we have pipelines to run acceptance tests that deploy the release using BOSH and run a test suite against those instances. Up until jammy (which is still the currently used stemcell) this is a shaky construct of endlessly nested containers which heavily relies on cgroupsv1 and the docker-cpi. The two main reasons for this setup are 1) execution time which is (relatively) low since containers can be spawned almost instantly and 2) costs as we can run the entire thing on a single VM as long as we restrict the CI to only run one job at a time.

As for the solution space:

• docker-cpi would need a re-work to support cgroupsv2
• warden-cpi should support cgroupsv2 but is lacking some of the ergonomics of the docker-cpi (not sure how big of a deal that is)
• vbox-cpi is a different solution that would still allow single-host environments for testing but uses actual VMs.

For the VBox CPI there was the idea from @ZPascal to rework it to not rely on CLI commands being exec'd and rather call some API. My suggestion would be to move to libvirt if we take that invest as it supports the most commonly used hypervisors (including VBox). There is also a linux container driver that might be able to replace the current warden / docker implementations.

There also seems to be an interest in making local environments more feasible as well as reducing the costs of the various CI systems that have been migrated to the CFF owned GCP accounts. Given that those three options are all (more or less) trying to solve the same issue it would probably make sense to focus on one scenario and officially support that.

If we can make a libvirt-based CPI work including the container driver that would probably be the most all-in-one solution we can get, but this a rather uninformed opinion so take it with a grain of salt :)

@beyhan it might make sense to move this discussion somewhere more central given the scope of what I'm describing here. If you want me to open a different issue for the discussion lmk.

Edit: yes, GCP does support nested virtualization

[wayneeseguin]

I'm going to take a crack at refactoring this project to use cgroupsv2 as my weekend project to see what I can come up with. I'll fork & create a branch and PR it if I can get it to work. Any extra contextual information is appreciated. You can also DM me in CF Slack.

[wayneeseguin]

Status update:
I've managed thus far to have it able to do tests/run.sh in a very good and repeatable manner using cgroupsv2 for jammy stemcells. I tried really really hard to make it work with Docker Desktop on OSX and decided to maybe revisit that at a later date 😅. I then went on to testing on a Jammy linux host and now have all expanded tests and integration passing and working. I am currently fighting with trying to get noble stemcells working... to which I mght need to take a mental sanity break from in a bit 🤪.

Either way I'll push up my changes to a branch on my fork and send a PR for comments and ideas for improvement.

I did update the way to approach running the tests, I now have a make test and make clean for cycling which I personally find easier. For stemcells can change which one via:

STEMCELL_OS=jammy make test # will query for latest version
STEMCELL_OS=noble make test # will query for latest version
STEMCELL_OS=noble STEMCELL_VERSION=1.571 make test

I'm also trying to make different options configurable (mostly through env vars) instead of hard coded for easy overrides.
🤞

[wayneeseguin]

Now trying to get the noble stemcell working, I've made it past booting the director:

-----> Sun Jun  1 00:10:22 UTC 2025: Create env
       Getting latest ubuntu-noble stemcell version...
       Latest version: 1.2
       Using stemcell: ubuntu-noble v1.2
       SHA1: 4670d8c7f62e54cb7a210be16760fe6b33adeeb2
Deployment manifest: '/stemcell/workspace/bosh-deployment/bosh.yml'
Deployment state: 'state.json'

Started validating
  Downloading release 'bosh'... Skipped [Found in local cache] (00:00:00)
  Validating release 'bosh'... Finished (00:00:00)
  Downloading release 'bpm'... Skipped [Found in local cache] (00:00:00)
  Validating release 'bpm'... Finished (00:00:00)
  Validating release 'bosh-docker-cpi'... Finished (00:00:01)
  Downloading release 'os-conf'... Skipped [Found in local cache] (00:00:00)
  Validating release 'os-conf'... Finished (00:00:00)
  Validating cpi release... Finished (00:00:00)
  Validating deployment manifest... Finished (00:00:00)
  Downloading stemcell... Skipped [Found in local cache] (00:00:00)
  Validating stemcell... Finished (00:00:05)
Finished validating (00:00:14)

Started installing CPI
  Compiling package 'golang-1-darwin/a98f303681628a352430c6a1c348bedc197ab11b60420b1d159f9966f75ad195'... Finished (00:00:12)
  Compiling package 'golang-1-linux/e75c69bba41945ce0ae901b68cdcb641b624f373fb0f63f282acc8d138ca27d2'... Finished (00:00:12)
  Compiling package 'docker_cpi/ee0aea16eabd46c5afe8ad412183a6eb408a774b5f753e8536c824a28b46374f'... Finished (00:00:20)
  Installing packages... Finished (00:00:04)
  Rendering job templates... Finished (00:00:00)
  Installing job 'docker_cpi'... Finished (00:00:00)
Finished installing CPI (00:00:50)

Uploading stemcell 'bosh-warden-boshlite-ubuntu-noble/1.2'... Finished (00:00:24)

Started deploying
  Creating VM for instance 'bosh/0' from stemcell 'bosh.io/stemcells:img-db56d064-ddc7-4f32-7856-2ea052b25d69'... Finished (00:00:00)
  Waiting for the agent on VM 'c-b3c81128-5d52-4bb6-5173-360d15911d05' to be ready... Finished (00:00:11)
  Creating disk... Finished (00:00:00)
  Attaching disk 'vol-8cde644b-efbb-496d-6aaf-b7a7c0cdf3b2' to VM 'c-b3c81128-5d52-4bb6-5173-360d15911d05'... Finished (00:00:13)
  Rendering job templates... Finished (00:00:05)
  Compiling package 'golang-1-linux/d742dc5eebc66113a186db2c1446ff5d646524cc798d1e74a56b684cffc53d35'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'golang-1-darwin/a98f303681628a352430c6a1c348bedc197ab11b60420b1d159f9966f75ad195'... Finished (00:00:14)
  Compiling package 'golang-1-linux/e75c69bba41945ce0ae901b68cdcb641b624f373fb0f63f282acc8d138ca27d2'... Finished (00:00:14)
  Compiling package 'director-ruby-3.3/b4532b6dd687638f471b788500016617d77cbe1e24b562355c93a555faaa52c4'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'mysql/5d3b888884916d278affc65eda58b6002c4f703f51af94213f50015cc0d00b79'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'libpq/a3d5cd5742dbfa1adf2d5c66c4bae26e9d42a0bf8d55a92d6f97213f08392c47'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'tini/bf4b98c300b214d8b83160db11b32f596da5bc236c22ae0c95fea75e7b3c4e05'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'bpm-runc/69545f216070f3affc3fb16f8d6a756d9e6249ddca9475cade498450f13c0110'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'docker_cpi/ee0aea16eabd46c5afe8ad412183a6eb408a774b5f753e8536c824a28b46374f'... Finished (00:00:21)
  Compiling package 'bosh-gcscli/5ba81129552e7f2f6673fa272142ffc427904f41e5df4610113d8ef5b1c5a785'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'postgres-15/f8fe8767df1b4025b5a09ffdb9eeccbb3fceda3c521d71b3ed2f2a8b56016ed4'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'health_monitor/fa9dedd307d8a1d0da2b7b6916699d3675698d7962adb3fe457902fbfa08ac1c'... Skipped [Package already compiled] (00:00:00)
  Compiling package 's3cli/dd97ff8f3696808dcf024b8a5b025b38ddc7b6501335d981ab67d524c5ac37a0'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'nats/560169976527f9cabcfad77869f71ff742c55d579e701712ca3d157b844276d9'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'azure-storage-cli/a62d7535fd6b3885136972a21fa63170bf23ad015a1e25c1817c39f3faa2c038'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'director/944dfccb9c1b105f1d8aa631c18d7e68206b961f7400b145e015003322c0a3fe'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'bpm/8caa1c2cd60e50a419b68b0060f81505ce0c7465827806aaf1f596319112c634'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'verify_multidigest/f85e7fe0b06771def8bcec800f80d3286d3711480aac2d3b261b5c653c10e2ad'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'postgres-13/16bcb38c167481d981288ed5b7dc1914d9a82e35ce8dcee693558e85179248b7'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'nginx/79d2479c4363ebac7a1d536f083f3c86b74ac016fcca24acad270517c9c6ed83'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'davcli/120c569896639f134bd1f1aeeb45c4f93b032f446f2a0a24769630274670f2d9'... Skipped [Package already compiled] (00:00:00)
  Updating instance 'bosh/0'... Finished (00:00:13)
  Waiting for instance 'bosh/0' to be running... Finished (00:02:39)
  Running the post-start scripts 'bosh/0'... Finished (00:00:01)
Finished deploying (00:04:17)

Cleaning up rendered CPI jobs... Finished (00:00:00)

Succeeded

Now working on debugging the upload stemcell:

-----> Sun Jun  1 00:16:10 UTC 2025: Upload stemcell
       Getting SHA1 for ubuntu-noble v1.2...
       SHA1: 4670d8c7f62e54cb7a210be16760fe6b33adeeb2
Using environment '10.245.0.11' as client 'admin'

Task 1

Task 1 | 00:16:11 | Update stemcell: Downloading remote stemcell (00:00:27)
Task 1 | 00:16:38 | Update stemcell: Verifying remote stemcell (00:00:02)
Task 1 | 00:16:40 | Update stemcell: Extracting stemcell archive (00:00:06)
Task 1 | 00:16:46 | Update stemcell: Verifying stemcell manifest (00:00:00)
Task 1 | 00:16:46 | Update stemcell: Checking if this stemcell already exists (00:00:00)
Task 1 | 00:16:46 | Update stemcell: Uploading stemcell bosh-warden-boshlite-ubuntu-noble/1.2 to the cloud (00:00:00)
                  L Error: CPI error 'Bosh::Clouds::CloudError' with message 'Importing stemcell from '/var/vcap/data/director/tmp/stemcell20250601-7627-42v672/image': Starting image import: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?' in 'create_stemcell' CPI method (CPI request ID: 'cpi-629774')
Task 1 | 00:16:46 | Error: CPI error 'Bosh::Clouds::CloudError' with message 'Importing stemcell from '/var/vcap/data/director/tmp/stemcell20250601-7627-42v672/image': Starting image import: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?' in 'create_stemcell' CPI method (CPI request ID: 'cpi-629774')

Task 1 Started  Sun Jun  1 00:16:11 UTC 2025
Task 1 Finished Sun Jun  1 00:16:46 UTC 2025
Task 1 Duration 00:00:35
Task 1 error

Uploading remote stemcell 'https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-noble?v=1.2':
  Expected task '1' to succeed but state is 'error'

Exit code 1
make: *** [Makefile:28: test] Error 1

[wayneeseguin]

Progress...

-----> Sun Jun  1 01:07:27 UTC 2025: Upload stemcell
       Getting SHA1 for ubuntu-noble v1.2...
       SHA1: 4670d8c7f62e54cb7a210be16760fe6b33adeeb2
Using environment '10.245.0.11' as client 'admin'

Task 1

Task 1 | 01:07:28 | Update stemcell: Downloading remote stemcell (00:00:27)
Task 1 | 01:07:55 | Update stemcell: Verifying remote stemcell (00:00:02)
Task 1 | 01:07:57 | Update stemcell: Extracting stemcell archive (00:00:05)
Task 1 | 01:08:03 | Update stemcell: Verifying stemcell manifest (00:00:00)
Task 1 | 01:08:03 | Update stemcell: Checking if this stemcell already exists (00:00:00)
Task 1 | 01:08:03 | Update stemcell: Uploading stemcell bosh-warden-boshlite-ubuntu-noble/1.2 to the cloud (00:00:23)
Task 1 | 01:08:26 | Update stemcell: Save stemcell bosh-warden-boshlite-ubuntu-noble/1.2 (bosh.io/stemcells:img-34c1da7f-37bb-4cd0-57aa-a7b78752761a) (00:00:00)

Task 1 Started  Sun Jun  1 01:07:28 UTC 2025
Task 1 Finished Sun Jun  1 01:08:26 UTC 2025
Task 1 Duration 00:00:58
Task 1 done

Succeeded

[wayneeseguin]

OK so new day new adventure, ran into an attach_disk error on the zookeeper vms deployment, digging in to debug that now.

[wayneeseguin]

The Zookeeper release (version 0.0.10) was compiled with Go 1.10 and is experiencing compatibility issues with the Ubuntu Noble stemcell. This shows up as an authentication issue, but is actually a connection/protocol issue that the old Go client is misinterpreting... so I'm going to skip that part of the test can be re-enabled later.

[wayneeseguin]

I now have a new Ubuntu 24.04 linux host that I'm going to run the tests on for both stemcells.

[beyhan]

@beyhan and I spoke a about this offline, I will first give a bit more context on why we have raised this issue:

As part of the haproxy-boshrelease we have pipelines to run acceptance tests that deploy the release using BOSH and run a test suite against those instances. Up until jammy (which is still the currently used stemcell) this is a shaky construct of endlessly nested containers which heavily relies on cgroupsv1 and the docker-cpi. The two main reasons for this setup are 1) execution time which is (relatively) low since containers can be spawned almost instantly and 2) costs as we can run the entire thing on a single VM as long as we restrict the CI to only run one job at a time.

As for the solution space:

• docker-cpi would need a re-work to support cgroupsv2
• warden-cpi should support cgroupsv2 but is lacking some of the ergonomics of the docker-cpi (not sure how big of a deal that is)
• vbox-cpi is a different solution that would still allow single-host environments for testing but uses actual VMs.

For the VBox CPI there was the idea from @ZPascal to rework it to not rely on CLI commands being exec'd and rather call some API. My suggestion would be to move to libvirt if we take that invest as it supports the most commonly used hypervisors (including VBox). There is also a linux container driver that might be able to replace the current warden / docker implementations.

There also seems to be an interest in making local environments more feasible as well as reducing the costs of the various CI systems that have been migrated to the CFF owned GCP accounts. Given that those three options are all (more or less) trying to solve the same issue it would probably make sense to focus on one scenario and officially support that.

If we can make a libvirt-based CPI work including the container driver that would probably be the most all-in-one solution we can get, but this a rather uninformed opinion so take it with a grain of salt :)

@beyhan it might make sense to move this discussion somewhere more central given the scope of what I'm describing here. If you want me to open a different issue for the discussion lmk.

Edit: yes, GCP does support nested virtualization

The progress on the cgroupv2 support reported by @wayneeseguin looks promising. Thank you @wayneeseguin for that! @maxmoehl regarding the idea to use libvirt as abstraction library for managing VMs or containers I would suggest to create an issue in vbox-cpi about switching to libvirt. I think, if this gets approached it makes sense to try first with one CPI and then think about other CPIs if it worked well.

[wayneeseguin]

@beyhan I'll do a PR either tonight after work or tomorrow, it worked fully for cgroupsv2 from a jammy host for both jammy and noble. I now have a noble host I'm trying to debug the tests with. A good thing is it caught dependency assumptions since it was a fresh host.