GH-47798: [CI][Packaging] Enable reproducible builds for Linux packages #47864

raulcd · 2025-10-20T10:45:48Z

TBD

This is just a testing PR at the moment to validate a CI job. There's still work to be done.

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

This PR includes breaking changes to public APIs. (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)

This PR contains a "Critical Fix". (If the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld), please provide explanation. If not, you can remove this.)

…packages

…porarily and skip rat check on patch

github-actions · 2025-10-20T10:46:22Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

…or reproducible builds

…ile even though it is wrong. Just testing purposes

…when HOME is modified

raulcd

This is currently building the artifacts 3 times, 2 via reprotest to validate reproducibility and 1 for the final artifact generated. Similar to when we run reprotest for our source code but on the Linux Packages case it obviously takes a really long time, 2h 30m on the debian-trixie-amd64 job that is running reprotest:
https://github.com/apache/arrow/actions/runs/18685243685/job/53276081878

Should the approach be to run reprotest always? It potentially could be interesting to disable REPROTEST on PR checks and enable it manually if necessary?

There is still a lot of work to be done here, apart from fixing reproducible builds when build_path variant is used and adding the requirements to other debian packages and rpm.

raulcd · 2025-10-21T16:32:18Z

dev/tasks/linux-packages/apt/build.sh

 export DEB_BUILD_OPTIONS
 df -h
+if [ "${REPROTEST:-no}" = "yes" ]; then
+  run reprotest --verbosity 2 --vary=-kernel,-fileordering,-domain_host,-build_path -s .. ./reprotest.sh **.deb


I am currently investigating why if build_path is exercised, basically applying the following diff:

Suggested change

run reprotest --verbosity 2 --vary=-kernel,-fileordering,-domain_host,-build_path -s .. ./reprotest.sh **.deb

run reprotest --verbosity 2 --vary=-kernel,-fileordering,-domain_host -s .. ./reprotest.sh **.deb

The binaries generated (**.deb) are not reproducible, I've tried different approaches like:

diff --git a/dev/tasks/linux-packages/apache-arrow/debian/rules b/dev/tasks/linux-packages/apache-arrow/debian/rules index 19dba393b1..17ef34fc4b 100755 --- a/dev/tasks/linux-packages/apache-arrow/debian/rules +++ b/dev/tasks/linux-packages/apache-arrow/debian/rules @@ -6,7 +6,7 @@ # This has to be exported to make some magic below work. export DH_OPTIONS -export DEB_BUILD_MAINT_OPTIONS=reproducible=-timeless +export DEB_BUILD_MAINT_OPTIONS= hardening=+all reproducible=-timeless,+fixfilepath BUILD_TYPE=relwithdebinfo @@ -31,6 +31,7 @@ override_dh_auto_configure: --builddirectory=cpp_build \ --buildsystem=cmake+ninja \ -- \ + $(shell dpkg-buildflags --export=configure) \ -DARROW_AZURE=$${ARROW_AZURE} \ -DARROW_BUILD_UTILITIES=ON \ -DARROW_COMPUTE=ON \

but no luck so far. More info about build_path in reproducible builds:
https://reproducible-builds.org/docs/build-path/

kou · 2025-10-29T08:14:18Z

How about running reprotest on host not in Docker container something like the following?

diff --git a/.github/workflows/package_linux.yml b/.github/workflows/package_linux.yml
index ba86389428..ead6faf525 100644
--- a/.github/workflows/package_linux.yml
+++ b/.github/workflows/package_linux.yml
@@ -229,10 +229,11 @@ jobs:
             release_candidate.yml
       - name: Build
         run: |
-          pushd dev/tasks/linux-packages
-          rake docker:pull || :
-          rake --trace ${TASK_NAMESPACE}:build BUILD_DIR=build
-          popd
+          rake -C dev/tasks/linux-packages docker:pull || :
+          BUILD_DIR=dev/tasks/linux-packages/build \
+            reprotest \
+              -c "rake -C dev/tasks/linux-packages --trace ${TASK_NAMESPACE}:build " \
+              . \ "dev/tasks/linux-packages/*/${TASK_NAMESPACE}/repositories/**/*.*"
       - name: Docker Push
         continue-on-error: true
         if: >-
diff --git a/dev/tasks/linux-packages/apache-arrow-apt-source/debian/rules b/dev/tasks/linux-packages/apache-arrow-apt-source/debian/rules
index 1e3be48c31..2a3c14c558 100755
--- a/dev/tasks/linux-packages/apache-arrow-apt-source/debian/rules
+++ b/dev/tasks/linux-packages/apache-arrow-apt-source/debian/rules
@@ -12,10 +12,12 @@ export DH_OPTIONS
 override_dh_auto_build:
 	gpg \
 	  --no-default-keyring \
+	  --homedir /tmp \
 	  --keyring ./apache-arrow-apt-source.kbx \
 	  --import KEYS
 	gpg \
 	  --no-default-keyring \
+	  --homedir /tmp \
 	  --keyring ./apache-arrow-apt-source.kbx \
 	  --armor \
 	  --export > apache-arrow-apt-source.asc
diff --git a/dev/tasks/linux-packages/apache-arrow/apt/debian-trixie/Dockerfile b/dev/tasks/linux-packages/apache-arrow/apt/debian-trixie/Dockerfile
index 257d005656..3c3c3a3ad9 100644
--- a/dev/tasks/linux-packages/apache-arrow/apt/debian-trixie/Dockerfile
+++ b/dev/tasks/linux-packages/apache-arrow/apt/debian-trixie/Dockerfile
@@ -39,6 +39,7 @@ RUN \
   apt install -y -V ${quiet} \
     base-files \
     build-essential \
+    ccache \
     clang \
     cmake \
     debhelper \
diff --git a/dev/tasks/linux-packages/apt/build.sh b/dev/tasks/linux-packages/apt/build.sh
index bc4c61e622..aa7ed976aa 100755
--- a/dev/tasks/linux-packages/apt/build.sh
+++ b/dev/tasks/linux-packages/apt/build.sh
@@ -48,8 +48,9 @@ architecture=$(dpkg-architecture -q DEB_BUILD_ARCH)
 debuild_options=()
 dpkg_buildpackage_options=(-us -uc)
 
-run mkdir -p /build
-run cd /build
+build_root_dir="/build"
+run mkdir -p "${build_root_dir}"
+run pushd "${build_root_dir}"
 find . -not -path ./ccache -a -not -path "./ccache/*" -delete
 if which ccache > /dev/null 2>&1; then
   export CCACHE_COMPILERCHECK=content
@@ -67,6 +68,8 @@ if which ccache > /dev/null 2>&1; then
     debuild_options+=(--prepend-path=/usr/lib/ccache)
   fi
 fi
+build_dir=$(mktemp --directory --tmpdir="${build_root_dir}" package.XXXXX)
+run pushd "${build_dir}"
 run cp /host/tmp/${PACKAGE}-${VERSION}.tar.gz \
   ${PACKAGE}_${VERSION}.orig.tar.gz
 run tar xfz ${PACKAGE}_${VERSION}.orig.tar.gz
@@ -80,7 +83,7 @@ case "${VERSION}" in
         ${PACKAGE}-${VERSION}
     ;;
 esac
-run cd ${PACKAGE}-${VERSION}/
+run pushd ${PACKAGE}-${VERSION}/
 platform="${distribution}-${code_name}"
 if [ -d "/host/tmp/debian.${platform}-${architecture}" ]; then
   run cp -rp "/host/tmp/debian.${platform}-${architecture}" debian
@@ -102,7 +105,7 @@ df -h
 if which ccache > /dev/null 2>&1; then
   ccache --show-stats --verbose || :
 fi
-run cd -
+run popd
 
 repositories="/host/repositories"
 package_initial=$(echo "${PACKAGE}" | sed -e 's/\(.\).*/\1/')
@@ -116,3 +119,7 @@ run \
   -exec cp '{}' "${pool_dir}/" ';'
 
 run chown -R "$(stat --format "%u:%g" "${repositories}")" "${repositories}"
+run find "${repositories}"
+
+run popd
+rm -rf "${build_dir}"
diff --git a/dev/tasks/linux-packages/package-task.rb b/dev/tasks/linux-packages/package-task.rb
index 4096c89463..d964a52dd3 100644
--- a/dev/tasks/linux-packages/package-task.rb
+++ b/dev/tasks/linux-packages/package-task.rb
@@ -150,7 +150,9 @@ class PackageTask
     end
     pass_through_env_names = [
       "DEB_BUILD_OPTIONS",
+      "HOME",
       "RPM_BUILD_NCPUS",
+      "TZ",
     ]
     pass_through_env_names.each do |name|
       value = ENV[name]
@@ -188,7 +190,7 @@ class PackageTask
     run_command_line << image
     run_command_line << "/host/build.sh" unless console
 
-    sh(*build_command_line)
+    sh(*build_command_line) if Dir.exist?(ENV["HOME"])
     sh(*run_command_line)
   end

(I'm trying this on local but it doesn't work yet. Sorry...)

raulcd · 2025-10-29T15:58:06Z

How about running reprotest on host not in Docker container something like the following?

This is fair, it would solve us requiring to patch reprotest as kernel is probably applicable with that and can fix some of the issues currently found.

I think we might also want to fix (maybe separately) a couple of issues around the build_path detected when running on the docker container, example using @basename@ instead of @filename@ on mkenums as suggested here: https://manpages.ubuntu.com/manpages/jammy/man1/glib-mkenums.1.html

I might open a PR for those separately.

Thanks for your help taking a look at this !

raulcd added 5 commits October 20, 2025 12:31

apacheGH-47798: [CI][Packaging] Enable reproducible builds for Linux …

cd694c0

…packages

Set reproducible environment variable only on debian-trixie-amd64 tem…

5dbc84e

…porarily and skip rat check on patch

Remove file pushed unintentionally

eab5aa7

Temporarily remove some linux packages

5290012

Add apt/reprotest.patch to .gitignore

ab460b0

github-actions bot added the awaiting committer review Awaiting committer review label Oct 20, 2025

raulcd added 6 commits October 20, 2025 12:53

Maybe using the correct environment variable name would do something :)

ee198d1

Commit missing SOURCE_DIR

dcee108

Double timeout as we have to build twice now and check architecture f…

609a52e

…or reproducible builds

Add Python3-venv to apache-arrow-apt-source/apt/debian-trixie/Dockerf…

81ba9ca

…ile even though it is wrong. Just testing purposes

Add missing faketime to Dockerfile

76661b5

Fix gpg homedir so we don't get differences with reproducible builds …

78b67ea

…when HOME is modified

raulcd commented Oct 21, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Oct 21, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

GH-47798: [CI][Packaging] Enable reproducible builds for Linux packages #47864

GH-47798: [CI][Packaging] Enable reproducible builds for Linux packages #47864

raulcd commented Oct 20, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Oct 20, 2025

Uh oh!

raulcd left a comment

Uh oh!

raulcd Oct 21, 2025

Uh oh!

kou commented Oct 29, 2025

Uh oh!

raulcd commented Oct 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	run reprotest --verbosity 2 --vary=-kernel,-fileordering,-domain_host,-build_path -s .. ./reprotest.sh **.deb
	run reprotest --verbosity 2 --vary=-kernel,-fileordering,-domain_host -s .. ./reprotest.sh **.deb

GH-47798: [CI][Packaging] Enable reproducible builds for Linux packages #47864

Are you sure you want to change the base?

GH-47798: [CI][Packaging] Enable reproducible builds for Linux packages #47864

Conversation

raulcd commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TBD

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Oct 20, 2025

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

raulcd Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

kou commented Oct 29, 2025

Uh oh!

raulcd commented Oct 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

raulcd commented Oct 20, 2025 •

edited

Loading