Conversation
Contributor
|
Contributor
Scan-Build Report
Bug Summary
Reports
|
||||||||||||||||||||||||||||||||||||
The x64-2.17 image's Dockerfile defaulted to openjdk:11-slim-buster. Debian 10 buster left even LTS support in June 2024 and its repositories are archived, so `apt-get update` against them now fails with "does not have a Release file" -- and rebuild-images.sh builds without --build-arg BASE_IMAGE whenever BASE_IMAGE_LIBC_2_17 is unset, which reaches that default. The body of the file is yum-based, so a Debian base cannot work there at all; the default now names what CI actually passes (centos:7), with the glibc 2.17 rationale for staying on an EOL distro recorded next to the vault mirror URLs it already needs. The amd64 build image moves to Debian 13 trixie, the current stable release. It compiles no shipped native code -- build:x64 uses the glibc 2.17 image -- so its glibc is not a customer-facing floor. The arm64 image stays on bullseye: it is where build:arm64 compiles the shipped linux-arm64 libjavaProfiler.so, so that glibc (2.31) is the runtime floor for arm64 customers and trixie would raise it to 2.41. Both variables now say which of the two they are and why. Also drop the `|| true` from the base image's package install, and with it the dead `apk` branch -- Alpine has its own Dockerfile.musl. As written, a total install failure produced a "successful" image whose tools were simply missing, surfacing later as an unrelated job failure. Verified against the real archives: buster's apt repositories fail as described, bullseye's still resolve, and the full package list installs on trixie with hexdump present (bsdmainutils is still a real package there). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rkennke
force-pushed
the
ci/bump-eol-base-images
branch
from
September 14, 2026 10:15
bd98bb8 to
5386860
Compare
Contributor
CI Test ResultsRun: #34863155419 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-09-14 15:53:19 UTC |
Release builds link -static-libstdc++ -static-libgcc, so the C++ runtime and
unwinder in the shipped libjavaProfiler.so come out of the build image's
toolchain. That makes the build image's support status a property of the
artifact, not just of CI: on CentOS 7 those objects can never receive another
security rebuild, and no gcc newer than devtoolset-10 was ever built for
aarch64 there (the dts-11 collection ships binutils, elfutils and its own
metapackage for aarch64, but no compiler).
AlmaLinux 8 with gcc-toolset-14 is supported until 2029 and publishes the
toolset for x86_64 and aarch64 alike, so one image definition now covers both
architectures with one compiler version, replacing the hand-rolled CentOS 7
image and its vault/Rocky mirror repositories. almalinux:8 is a multi-arch
manifest, so a single BASE_IMAGE_GLIBC serves both.
Measured on the real artifact, built in this image for both architectures:
max required symbol version GLIBC_2.27 (both x86_64 and aarch64)
symbols above 2.17 expf (2.27), getentropy (2.25)
dynamic dependencies libc, libdl, libm, libpthread, librt
GLIBCXX/CXXABI references none
So the runtime floor moves 2.17 -> 2.27, not to the base image's 2.28: symbol
versions are per-function. 2.27 keeps Ubuntu 18.04 (glibc 2.27) in support and
drops RHEL 7 (2.17), Amazon Linux 2 (2.26) and SLES 15 SP1 (2.26). Both
symbols are addressable with a .symver pin if a lower floor is needed later.
With the shipped arm64 build moved off the Debian image, that image no longer
sets a customer-facing floor, so it moves to trixie alongside amd64 -- which
also restores apt on the arm64 sanitizer jobs.
Names now describe the purpose rather than a version that has changed twice:
BUILD_IMAGE_{X64,ARM64}_GLIBC and BASE_IMAGE_GLIBC, with tag suffixes
{x64,arm64}-glibc-base. The variables still hold the current images until
rebuild-images.sh publishes the new tags.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rkennke
changed the title
The sanitizer jobs installed cmake, libgtest-dev, libgmock-dev, binutils, libc6-dbg and llvm in a before_script, which made every run depend on the distro archive still serving them, and which does not survive the move to trixie: sanitizer/asan_interface.h is no longer part of the llvm package there, so compiling linearAllocator.cpp fails with fatal error: 'sanitizer/asan_interface.h' file not found The header comes from libclang-rt-<n>-dev. The packages now live in the image instead, using the unversioned libclang-rt-dev metapackage so the right clang runtime is picked up on whatever release the image is built from. Dropping the before_script override also stops these jobs bypassing the pipeline default, so they get the CANCELLED guard and the maven proxy export that every other job already had. Verified on a locally built trixie image (clang 19.1.7, gtest 1.16.0), aarch64: buildGtestAsan and buildGtestTsan both compile, and all 62 asan and 62 tsan binaries pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pins from the rebuild in pipeline 137281788, which built every image from the definitions on this branch. Verified by pulling each digest: x64-glibc-base AlmaLinux 8.10, glibc 2.28, gcc 14.2.1 arm64-glibc-base AlmaLinux 8.10, glibc 2.28, gcc 14.2.1 x64-base Debian 13 trixie, glibc 2.41, clang 19.1.7 arm64-base Debian 13 trixie The musl, datadog-ci and benchmark images come from the same rebuild; their definitions did not change on this branch. These pins belong here rather than in the image-update PR the rebuild opened against main: main has no BUILD_IMAGE_*_GLIBC variables, so the two shipped-binary pins silently did nothing there, while BUILD_IMAGE_ARM64 -- which main still uses for build:arm64 and stresstest:arm64 -- would have moved the shipped arm64 binary onto a trixie image and its glibc 2.41 floor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
❌ 39 passed, 1 failed out of 40 configurations Test Matrix
Failure Detailsmusl-x64-openj9-jdk21Tracer+profiler: Links
|
rkennke
marked this pull request as ready for review
September 14, 2026 17:16
![datadog-datadog-prod-us1[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
More details
The CI jobs and rebuild scripts use consistent new image names. The Bullseye fallback already fails on the base branch, so it is not a regression from this PR.
🤖 Datadog Autotest · Commit 7bfc401 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7bfc40165c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Comment on lines
+1
to
+4
| # CI always passes BASE_IMAGE explicitly (OPENJDK_BASE_IMAGE / | ||
| # OPENJDK_BASE_IMAGE_ARM64), and the two arches deliberately differ -- see the | ||
| # comments on those variables. This default is the more conservative of the | ||
| # two so a hand-built image matches the arm64 glibc floor. |
There was a problem hiding this comment.
When this Dockerfile is built directly, or when .gitlab/scripts/rebuild-images.sh reaches its documented unset-variable fallback, BASE_IMAGE still resolves to the Bullseye digest on line 5 rather than the new Trixie image. The added comment is therefore incorrect—both CI architecture variables now use Trixie—and, because this commit also makes the package installation fatal, the fallback encounters the same unavailable Bullseye packages this change is intended to avoid. Update the default to the pinned Trixie base or require BASE_IMAGE explicitly.
AGENTS.md reference: AGENTS.md:L450-L454
Useful? React with 👍 / 👎.
| @@ -0,0 +1,37 @@ | |||
| # Build image for the shipped glibc binaries, both architectures. | |||
There was a problem hiding this comment.
This newly added Dockerfile has no Datadog copyright notice; the other non-deleted files modified by this commit likewise lack one. Add the required current-year Datadog notice to each touched file before merging.
AGENTS.md reference: AGENTS.md:L382-L384
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Spun out of the CI investigation on #786. Gets the GitLab pipeline off end-of-life base images, and moves the shipped binaries onto a toolchain that still receives security rebuilds.
Root cause of the current failures
bullseye-security's pool files have been pruned upstream while the index still advertises them — Debian 11 left LTS on 2026-08-31. Soapt-get updatesucceeds andapt-get installthen 404s on the individual.debs:Reproduced outside CI, straight from Fastly with no proxy in the path, so it is neither the fabric egress gateway nor a stale mirror. The
+deb11uNNversions and glibc 2.31 also identify the running image as bullseye.Bullseye cannot be rescued in place:
bullseye-securityis not published onarchive.debian.orgunder either its own name or the olderbullseye/updatesname, and the main archive alone leaves held broken packages. A bullseye image can no longer even be rebuilt, since its ownapt-get installlayer fails. Verified: the job's exact package list installs cleanly on trixie.Why the shipped-binary image matters more than it looks
Release builds link
-static-libstdc++ -static-libgcc(ConfigurationPresets.kt:121-122), so the C++ runtime and unwinder in the shippedlibjavaProfiler.socome out of the build image's toolchain. The build image's support status is therefore a property of the artifact, not just of CI.On CentOS 7 those objects can never get another security rebuild. Worse for consistency: no gcc newer than devtoolset-10 was ever built for aarch64 anywhere in the EL7 ecosystem — the dts-11 collection ships
binutils2.36.1,elfutils,dwzand its owntoolchainmetapackage for aarch64, but no compiler. Confirmed against CentOS vault altarch, Oracle's SCL repo and the CentOS SIG buildlogs. Oracle Linux 7 is the best-maintained EL7 derivative (live CDN, both arches, working collections) but its public repos sit at the same 7.9 package levels, with later content behind an ELS subscription.Changes
1. Shipped binaries: AlmaLinux 8 + gcc-toolset-14, both arches — new
.gitlab/base/el8/Dockerfile, replacing.gitlab/base/centos7/Dockerfileand itsvault.centos.org/ Rocky-mirror repository plumbing. Supported until 2029, and the toolset is published for x86_64 and aarch64 alike, so one image definition covers both architectures with one compiler version.almalinux:8is a multi-arch OCI index (amd64, arm64, ppc64le, s390x), so a singleBASE_IMAGE_GLIBCdigest serves both.2. Test/tooling images: Debian 13 trixie, both arches. With the shipped arm64 build moved off the Debian image, that image no longer sets a customer-facing floor, so it moves to current stable alongside amd64 — which is what restores apt on the arm64 sanitizer jobs.
3. The archived buster default is gone with
centos7/Dockerfile. It defaulted toopenjdk:11-slim-buster; buster's repos are archived (does not have a Release file) andrebuild-images.shreaches that default whenever the base-image variable is unset, since it onlylog_warns.4.
.gitlab/base/Dockerfile: no more swallowed install failures. Both theapt-get installandapk addlines ended in|| true, so a total package-install failure produced a "successful" image whose tools were simply missing. Theapkbranch was dead — Alpine/musl has its ownDockerfile.musl.5. Names describe purpose, not a version that has now changed twice:
BUILD_IMAGE_{X64,ARM64}_GLIBC,BASE_IMAGE_GLIBC, tag suffixes{x64,arm64}-glibc-base.The glibc floor: measured, not assumed
The real profiler was built inside the new image on genuine x86_64 and aarch64, and the artifact inspected:
GLIBC_2.27GLIBC_2.27expf(2.27),getentropy(2.25)GLIBCXX/CXXABIreferencesThe floor moves 2.17 → 2.27, not to the base image's 2.28 — symbol versions are per-function, so building on glibc 2.28 does not make the artifact require 2.28. No
libstdc++dynamic dependency, confirming the static link still works with the toolset's ownlibstdc++.a.Amazon Linux 2 is the only meaningful casualty, and it is not forced: both binding symbols are addressable with a
.symverpin (expf→ an older version,getentropy→ a directSYS_getrandomsyscall) if a lower floor is needed. Do not "fix"expfby switching to doubleexp, which carries its ownGLIBC_2.29version.This floor change needs a decision, and is the one thing here that reaches customers. Everything else is CI-internal.
Images rebuilt and repinned
The images were rebuilt from this branch's definitions in pipeline 137281788 and the digests are pinned here. Each was verified by pulling it:
x64-glibc-base,arm64-glibc-basex64-base,arm64-baseThe musl, datadog-ci and benchmark images come from the same rebuild; their definitions are unchanged on this branch.
For the record on why the pins are here rather than in the image-update PR that rebuild opened against
main(#791, closed):mainhas noBUILD_IMAGE_*_GLIBCvariables, so the two shipped-binary pins silently did nothing there, whileBUILD_IMAGE_ARM64— whichmainstill uses forbuild:arm64andstresstest:arm64— would have moved the shipped arm64 binary onto a trixie image and its glibc 2.41 floor.Worth knowing for the future: the pins live in this repo, and every rebuild creates new pipeline-ID-prefixed tags rather than overwriting old ones, so rebuilding affects no branch until a repin is merged.
Verified locally
.gitlab/base/el8/Dockerfilebuilds for both platforms;gcc-toolset-14provideslibstdc++.a/libgcc.a, the profiler's exact link flags succeed, andjqneeds no EPEL../gradlew :ddprof-lib:buildReleasesucceeds inside the image on both architectures, and the measurements above come from those artifacts.Also noticed, not changed
DOCKER_IMAGE: …/images/docker:24.0.4-gbi-focal(images.yml) is Ubuntu 20.04, past standard support since April 2025. Left alone because I cannot enumerate replacement tags in that ECR registry..gitlab/fuzzing/.gitlab-ci.yml:20and.gitlab/jdk-integration/.gitlab-ci.yml:32alsoapt-get installat job time, so they depend on a live archive on every run, exactly as the sanitizer jobs do. Moving those packages into the images would make the pipeline independent of archive availability.ubuntu-latest/ubuntu-22.04runners andalpine:3.23containers are all still supported.🤖 Generated with Claude Code