Skip to content

Carry buildkit's environment, daemon flags, and socket to the builder - #2188

Closed
MayCXC wants to merge 5 commits into
apple:mainfrom
MayCXC:remote-builder
Closed

Carry buildkit's environment, daemon flags, and socket to the builder#2188
MayCXC wants to merge 5 commits into
apple:mainfrom
MayCXC:remote-builder

Conversation

@MayCXC

@MayCXC MayCXC commented Aug 27, 2026

Copy link
Copy Markdown

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update

Motivation and Context

Closes #2187. Closes #2186.

Exposing buildkit. The builder wraps buildkit and exposed almost none of it: two color variables crossed from the caller's environment into the container, the daemon's flags were unreachable, and its socket never left the VM.

Every BUILDKIT_ variable in the caller's environment now rides into the builder, so BUILDKIT_HOST points the shim's client at a daemon of the operator's choosing, the BUILDKIT_TLS set carries that address's credentials, and whatever buildkit documents next needs no new plumbing; NO_COLOR rides along as the conventional outlier the color handling already honored, and a change to any of them recreates the builder the way the color variables always did.

--buildkitd-flags takes one string of daemon flags, buildx's own contract for the same name, handed to the shim after -- where it passes them to buildkitd verbatim. --publish-buildkit-socket puts the daemon's socket on the host through the runtime's socket publishing, where buildctl and buildx dial it directly, the daemon-socket convention Docker Desktop, colima, and podman machine follow on macOS; remote machines reach the same socket through buildkit's own ssh scheme.

The drift comparison. The builder's process environment mixes the image's own variables with the ones start injects; moby/buildkit's image carries BUILDKIT_SETUP_CGROUPV2_ROOT, so a start that reads the whole environment back sees variables it never injected and recreates a builder that already matches its target, and concurrent builds then race one another's delete and create cycles. The injected set rides a label on the container record, stamped at create and compared on the next start, the way docker compose stamps com.docker.compose.config-hash on the services it manages.

A builder replaced mid-flight interrupts every build sharing it, so the replacement names its reason: the drifted settings print to stderr beside the progress output before the stop.

The concurrent delete-and-create racing above is the same window #2174 serializes; the two compose, and this removes most of the occasions for the race rather than the race itself.

Testing

  • Tested locally
  • Added/updated tests
  • Added/updated docs

One builder start carries a BUILDKIT_ environment variable, daemon flags after --buildkitd-flags, and --publish-buildkit-socket. The test reads each where its consumer does: the shim's environment at /proc/1/environ, buildkitd's argv for the passed flags, and the host filesystem for the published socket, then runs a build with all three applied.

A second case covers the drift decision: a start whose inputs match the running builder returns it untouched; a start whose managed environment differs replaces it. A file written into the builder's filesystem tells the cases apart, surviving a kept builder and absent from a recreated one.

Integration suite: 397 passed. Unit suite: 772 passed. make fmt, make check clean.

MayCXC added 5 commits August 27, 2026 20:16
The builder wraps buildkit and exposed almost none of it: two color
variables crossed from the caller's environment into the container, the
daemon's flags were unreachable, and its socket never left the VM.

Every BUILDKIT_ variable in the caller's environment now rides into the
builder, so BUILDKIT_HOST points the shim's client at a daemon of the
operator's choosing, the BUILDKIT_TLS set carries that address's
credentials, and whatever buildkit documents next needs no new plumbing;
NO_COLOR rides along as the conventional outlier the color handling
already honored, and a change to any of them recreates the builder the
way the color variables always did.

--buildkitd-flags takes one string of daemon flags, buildx's own
contract for the same name, handed to the shim after -- where it passes
them to buildkitd verbatim. --publish-buildkit-socket puts the daemon's
socket on the host through the runtime's socket publishing, where
buildctl and buildx dial it directly, the daemon-socket convention
Docker Desktop, colima, and podman machine follow on macOS; remote
machines reach the same socket through buildkit's own ssh scheme.
One builder start carries a BUILDKIT_ environment variable, daemon
flags after --buildkitd-flags, and --publish-buildkit-socket. The test
reads each where its consumer does: the shim's environment at
/proc/1/environ, buildkitd's argv for the passed flags, and the host
filesystem for the published socket, then runs a build with all three
applied.
The builder's process environment mixes the image's own variables with
the ones start injects; moby/buildkit's image carries
BUILDKIT_SETUP_CGROUPV2_ROOT, so a start that reads the whole
environment back sees variables it never injected and recreates a
builder that already matches its target, and concurrent builds then
race one another's delete and create cycles. The injected set rides a
label on the container record, stamped at create and compared on the
next start, the way docker compose stamps
com.docker.compose.config-hash on the services it manages.
A start whose inputs match the running builder returns it untouched; a
start whose managed environment differs replaces it. A file written
into the builder's filesystem tells the cases apart: it survives a
kept builder and is absent from a recreated one.
A builder replaced mid-flight interrupts every build sharing it, so
the replacement names its reason: the drifted settings print to
stderr beside the progress output before the stop.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants