Error response from daemon: client version 1.24 is too old

[seballiot]

Describe the bug

Error happening since the 11 nov 2025 when using the gitlab-runner with AWS spot instance

Used config :

source  = "cattle-ops/gitlab-runner/aws"
version = "9.5.0"

runner_instance = {
    ...
    type                        = "t3a.medium"
  }

AMI of workers : ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-20251112

To Reproduce

The error

Preparing the "docker+machine" executor
ERROR: Failed to remove network for build
ERROR: Preparation failed: Error response from daemon: client version 1.42 is too old. Minimum supported API version is 1.44, please upgrade your client to a newer version (docker.go:779:0s)

Additional context

[X-Guardian]

There has been a new release of Docker, v29 which is causing this. docker-machine will download the latest version of Docker by default, if it is not pre-installed in the runner-worker AMI. Ref: https://gitlab.com/cki-project/mirror/docker-machine/-/issues/5.

Workaround/fix is to create your own AMI for the runner-worker with the Docker components pre-installed. These are the component versions that were live before the 11th Nov:

DOCKER_VERSION="5:28.5.2-*"
CONTAINERD_VERSION="1.7.29-*"
DOCKER_COMPOSE_VERSION="2.40.3-*"
DOCKER_CE_ROOTLESS_EXTRAS_VERSION="5:28.5.2-*"
DOCKER_BUILDX_VERSION="0.29.1-*"
DOCKER_MODEL_VERSION="0.1.44-*"

[seballiot]

Our workaround was to use Docker v28 from rancher/install-docker with the option engine-install-url

runner_worker_docker_machine_ec2_options = [
  "engine-install-url='https://releases.rancher.com/install-docker/28.2.1.sh'"
]

[kayman-mk]

Yeah, updating dependencies at runtime. My personal horror. At midnight all pipelines started to fail.

I can confirm the workaround.