Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable Bzlmod for Bazel #19087

Closed
wants to merge 8 commits into from
Closed

Enable Bzlmod for Bazel #19087

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
33bcfa1
WIP: enable Bzlmod for Bazel
meteorcloudy Sep 1, 2023
0a4836a
Update lockfile
meteorcloudy Sep 6, 2023
b340398
Move all Maven dependencies to MODULE.bazel
meteorcloudy Sep 6, 2023
72f4a3a
Fix zlib BUILD file
meteorcloudy Sep 6, 2023
6527246
Update postsubmit.yml
meteorcloudy Sep 6, 2023
daa6e36
Update hash for bazel_tools in bootstrap build
meteorcloudy Sep 7, 2023
e070a9b
Fetch test_repos in the old way
meteorcloudy Sep 7, 2023
13f77ba
Fix bazel_determinisim_test
meteorcloudy Sep 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 71 additions & 111 deletions .bazelci/postsubmit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,52 +1,41 @@
---
matrix:
bzlmod_platforms: ["centos7_java11_devtoolset10", "ubuntu2004", "macos", "windows"]

tasks:
bzlmod_builds:
name: "Bazel Bzlmod build"
platform: ${{ bzlmod_platforms }}
build_flags:
- "--config=bzlmod"
build_targets:
- "//src:bazel"
test_targets:
- "//src/test/cpp/..."
centos7_java11_devtoolset10:
shell_commands:
- sed -i.bak -e 's/^# android_sdk_repository/android_sdk_repository/' -e 's/^#
android_ndk_repository/android_ndk_repository/' WORKSPACE
- rm -f WORKSPACE.bak
android_ndk_repository/android_ndk_repository/' WORKSPACE.bzlmod
- rm -f WORKSPACE.bzlmod.bak
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we creating this .bak in the first place if we delete it right away?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just for compatibility with macOS, where sed doesn't support in-place replacing without a .bak file.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this is nitpicking (so feel free to leave as is), but why not do this weird pattern only on Mac?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't know the original reason, but I guess it's just easier to keep them the same so you can search and edit all places just one time when you need some changes. We can clean this up in another PR if necessary.

- rm -rf $HOME/bazeltest
- mkdir $HOME/bazeltest
build_flags:
- "--test_env=TEST_INSTALL_BASE=$HOME/bazeltest/install_base"
- "--test_env=TEST_REPOSITORY_HOME=$OUTPUT_BASE/external"
- "--noremote_accept_cached"
- "--config=ci-linux"
meisterT marked this conversation as resolved.
Show resolved Hide resolved
build_targets:
- "//:bazel-distfile.zip"
- "//scripts/packages/debian:bazel-debian.deb"
- "//scripts/packages:with-jdk/install.sh"
- "//src:bazel"
- "//src:bazel_jdk_minimal"
- "//src:test_repos"
- "//src/main/java/..."
- "@test_repos//:srcs"
test_flags:
- "--sandbox_default_allow_network=false"
- "--sandbox_writable_path=$HOME/bazeltest"
- "--test_env=TEST_INSTALL_BASE=$HOME/bazeltest/install_base"
- "--test_env=TEST_REPOSITORY_HOME=$OUTPUT_BASE/external"
- "--config=ci-linux"
# Override REMOTE_NETWORK_ADDRESS since bazel_sandboxing_networking_test doesn't work on this platform
- "--test_env=REMOTE_NETWORK_ADDRESS="
test_targets:
- "//scripts/..."
- "//src/java_tools/..."
- "//src/main/starlark/tests/builtins_bzl/..."
- "//src/test/..."
- "//src/tools/execlog/..."
- "//src/tools/singlejar/..."
- "//src/tools/workspacelog/..."
- "//third_party/ijar/..."
- "//tools/android/..."
- "//tools/aquery_differ/..."
- "//tools/compliance/..."
- "//tools/python/..."
- "//tools/bash/..."
# Re-enable once fixed: https://github.com/bazelbuild/bazel/issues/8162
- "-//src/java_tools/buildjar/..."
- "-//src/java_tools/import_deps_checker/..."
Expand All @@ -68,31 +57,24 @@ tasks:
ubuntu1804:
shell_commands:
- sed -i.bak -e 's/^# android_sdk_repository/android_sdk_repository/' -e 's/^#
android_ndk_repository/android_ndk_repository/' WORKSPACE
- rm -f WORKSPACE.bak
android_ndk_repository/android_ndk_repository/' WORKSPACE.bzlmod
- rm -f WORKSPACE.bzlmod.bak
- rm -rf $HOME/bazeltest
- mkdir $HOME/bazeltest
build_flags:
- "--test_env=TEST_INSTALL_BASE=$HOME/bazeltest/install_base"
- "--test_env=TEST_REPOSITORY_HOME=$OUTPUT_BASE/external"
- "--test_env=REMOTE_NETWORK_ADDRESS=bazel.build:80"
- "--noremote_accept_cached"
- "--config=ci-linux"
build_targets:
- "//src:bazel"
- "//src:bazel_jdk_minimal"
- "//src:test_repos"
- "//src/main/java/..."
- "@test_repos//:srcs"
test_flags:
- "--sandbox_default_allow_network=false"
- "--sandbox_writable_path=$HOME/bazeltest"
- "--test_env=TEST_INSTALL_BASE=$HOME/bazeltest/install_base"
- "--test_env=TEST_REPOSITORY_HOME=$OUTPUT_BASE/external"
- "--config=ci-linux"
- "--test_tag_filters=-no_1804"
# Configure and enable tests that require access to the network.
- "--test_env=REMOTE_NETWORK_ADDRESS=bazel.build:80"
test_targets:
- "//scripts/..."
- "//src/java_tools/..."
- "//src/main/starlark/tests/builtins_bzl/..."
- "//src/test/..."
- "//src/tools/execlog/..."
- "//src/tools/singlejar/..."
Expand All @@ -101,6 +83,7 @@ tasks:
- "//tools/android/..."
- "//tools/aquery_differ/..."
- "//tools/python/..."
- "//tools/bash/..."
# Re-enable once fixed: https://github.com/bazelbuild/bazel/issues/8162
- "-//src/java_tools/import_deps_checker/..."
# https://github.com/bazelbuild/bazel/issues/18776
Expand All @@ -121,25 +104,22 @@ tasks:
name: "Clang"
shell_commands:
- sed -i.bak -e 's/^# android_sdk_repository/android_sdk_repository/' -e 's/^#
android_ndk_repository/android_ndk_repository/' WORKSPACE
- rm -f WORKSPACE.bak
android_ndk_repository/android_ndk_repository/' WORKSPACE.bzlmod
- rm -f WORKSPACE.bzlmod.bak
- rm -rf $HOME/bazeltest
- mkdir $HOME/bazeltest
build_flags:
- "--test_env=TEST_INSTALL_BASE=$HOME/bazeltest/install_base"
- "--test_env=TEST_REPOSITORY_HOME=$OUTPUT_BASE/external"
- "--noremote_accept_cached"
- "--config=ci-linux"
- "--features=-layering_check"
build_targets:
- "//src:bazel"
- "//src:bazel_jdk_minimal"
- "//src:test_repos"
- "//src/main/java/..."
- "@test_repos//:srcs"
test_flags:
- "--sandbox_default_allow_network=false"
- "--sandbox_writable_path=$HOME/bazeltest"
- "--test_env=TEST_INSTALL_BASE=$HOME/bazeltest/install_base"
- "--test_env=TEST_REPOSITORY_HOME=$OUTPUT_BASE/external"
- "--config=ci-linux"
- "--test_tag_filters=-no_1804"
- "--features=-layering_check"
test_targets:
- "//src/test/shell/bazel:cc_integration_test"
include_json_profile:
Expand All @@ -148,30 +128,23 @@ tasks:
ubuntu2004:
shell_commands:
- sed -i.bak -e 's/^# android_sdk_repository/android_sdk_repository/' -e 's/^#
android_ndk_repository/android_ndk_repository/' WORKSPACE
- rm -f WORKSPACE.bak
android_ndk_repository/android_ndk_repository/' WORKSPACE.bzlmod
- rm -f WORKSPACE.bzlmod.bak
- rm -rf $HOME/bazeltest
- mkdir $HOME/bazeltest
build_flags:
- "--test_env=TEST_INSTALL_BASE=$HOME/bazeltest/install_base"
- "--test_env=TEST_REPOSITORY_HOME=$OUTPUT_BASE/external"
- "--test_env=REMOTE_NETWORK_ADDRESS=bazel.build:80"
- "--noremote_accept_cached"
- "--config=ci-linux"
build_targets:
- "//src:bazel"
- "//src:bazel_jdk_minimal"
- "//src:test_repos"
- "//src/main/java/..."
- "@test_repos//:srcs"
test_flags:
- "--sandbox_default_allow_network=false"
- "--sandbox_writable_path=$HOME/bazeltest"
- "--test_env=TEST_INSTALL_BASE=$HOME/bazeltest/install_base"
- "--test_env=TEST_REPOSITORY_HOME=$OUTPUT_BASE/external"
# Configure and enable tests that require access to the network.
- "--test_env=REMOTE_NETWORK_ADDRESS=bazel.build:80"
- "--config=ci-linux"
test_targets:
- "//scripts/..."
- "//src/java_tools/..."
- "//src/main/starlark/tests/builtins_bzl/..."
- "//src/test/..."
- "//src/tools/execlog/..."
- "//src/tools/singlejar/..."
Expand All @@ -180,6 +153,7 @@ tasks:
- "//tools/android/..."
- "//tools/aquery_differ/..."
- "//tools/python/..."
- "//tools/bash/..."
# Re-enable once fixed: https://github.com/bazelbuild/bazel/issues/8162
- "-//src/java_tools/import_deps_checker/..."
# https://github.com/bazelbuild/bazel/issues/18776
Expand All @@ -195,33 +169,25 @@ tasks:
macos:
shell_commands:
- sed -i.bak -e 's/^# android_sdk_repository/android_sdk_repository/' -e 's/^#
android_ndk_repository/android_ndk_repository/' WORKSPACE
- rm -f WORKSPACE.bak
android_ndk_repository/android_ndk_repository/' WORKSPACE.bzlmod
- rm -f WORKSPACE.bzlmod.bak
- rm -rf $HOME/bazeltest
- mkdir $HOME/bazeltest
- ln -sf $OUTPUT_BASE/external $HOME/bazeltest/external
build_flags:
- "--test_env=TEST_INSTALL_BASE=$HOME/bazeltest/install_base"
- "--test_env=TEST_REPOSITORY_HOME=$OUTPUT_BASE/external"
- "--test_env=TEST_REPOSITORY_HOME=$HOME/bazeltest/external"
- "--test_env=REMOTE_NETWORK_ADDRESS=bazel.build:80"
- "--noremote_accept_cached"
- "--config=ci-macos"
build_targets:
- "//src:bazel"
- "//src:bazel_jdk_minimal"
- "//src:test_repos"
- "//src/main/java/..."
- "@test_repos//:srcs"
test_flags:
- "--sandbox_default_allow_network=false"
- "--sandbox_writable_path=$HOME/bazeltest"
- "--test_env=TEST_INSTALL_BASE=$HOME/bazeltest/install_base"
- "--test_env=TEST_REPOSITORY_HOME=$HOME/bazeltest/external"
# Configure and enable tests that require access to the network.
- "--test_env=REMOTE_NETWORK_ADDRESS=bazel.build:80"
- "--config=ci-macos"
# Fine tune the number of test jobs running in parallel to avoid timeout
- "--local_test_jobs=8"
test_targets:
- "//scripts/..."
- "//src/main/starlark/tests/builtins_bzl/..."
- "//src/test/..."
- "//src/tools/execlog/..."
- "//src/tools/singlejar/..."
Expand All @@ -230,6 +196,10 @@ tasks:
- "//tools/android/..."
- "//tools/aquery_differ/..."
- "//tools/python/..."
- "//tools/bash/..."
# Re-enable once fixed: https://github.com/bazelbuild/bazel/issues/8162
- "-//src/java_tools/buildjar/..."
- "-//src/java_tools/import_deps_checker/..."
# C++ coverage is not supported on macOS yet.
- "-//src/test/shell/bazel:bazel_cc_code_coverage_test"
# MacOS does not have cgroups so it can't support hardened sandbox
Expand Down Expand Up @@ -263,31 +233,23 @@ tasks:
macos_arm64:
shell_commands:
- sed -i.bak -e 's/^# android_sdk_repository/android_sdk_repository/' -e 's/^#
android_ndk_repository/android_ndk_repository/' WORKSPACE
- rm -f WORKSPACE.bak
android_ndk_repository/android_ndk_repository/' WORKSPACE.bzlmod
- rm -f WORKSPACE.bzlmod.bak
- rm -rf $HOME/bazeltest
- mkdir $HOME/bazeltest
- ln -sf $OUTPUT_BASE/external $HOME/bazeltest/external
build_flags:
- "--test_env=TEST_INSTALL_BASE=$HOME/bazeltest/install_base"
- "--test_env=TEST_REPOSITORY_HOME=$OUTPUT_BASE/external"
- "--test_env=TEST_REPOSITORY_HOME=$HOME/bazeltest/external"
- "--test_env=REMOTE_NETWORK_ADDRESS=bazel.build:80"
- "--noremote_accept_cached"
- "--config=ci-macos"
build_targets:
- "//src:bazel"
- "//src:bazel_jdk_minimal"
- "//src:test_repos"
- "//src/main/java/..."
- "@test_repos//:srcs"
test_flags:
- "--sandbox_default_allow_network=false"
- "--sandbox_writable_path=$HOME/bazeltest"
- "--test_env=TEST_INSTALL_BASE=$HOME/bazeltest/install_base"
- "--test_env=TEST_REPOSITORY_HOME=$HOME/bazeltest/external"
# Configure and enable tests that require access to the network.
- "--test_env=REMOTE_NETWORK_ADDRESS=bazel.build:80"
- "--config=ci-macos"
test_targets:
- "//scripts/..."
- "//src/main/starlark/tests/builtins_bzl/..."
- "//src/test/..."
- "//src/tools/execlog/..."
- "//src/tools/singlejar/..."
Expand All @@ -296,6 +258,10 @@ tasks:
- "//tools/android/..."
- "//tools/aquery_differ/..."
- "//tools/python/..."
- "//tools/bash/..."
# Re-enable once fixed: https://github.com/bazelbuild/bazel/issues/8162
- "-//src/java_tools/buildjar/..."
- "-//src/java_tools/import_deps_checker/..."
# C++ coverage is not supported on macOS yet.
- "-//src/test/shell/bazel:bazel_cc_code_coverage_test"
# MacOS does not have cgroups so it can't support hardened sandbox
Expand All @@ -319,32 +285,23 @@ tasks:
- build
- test
windows:
batch_commands:
- powershell -Command "(Get-Content WORKSPACE) -Replace '# android_', 'android_' | Set-Content WORKSPACE"
setup:
- mkdir C:\b
- mklink /J C:\b\bazeltest_external %OUTPUT_BASE:/=\%\external
batch_commands:
- powershell -Command "(Get-Content WORKSPACE.bzlmod) -Replace '# android_', 'android_' | Set-Content WORKSPACE.bzlmod"
build_flags:
- "--copt=-w"
- "--host_copt=-w"
- "--test_env=JAVA_HOME"
- "--test_env=TEST_INSTALL_BASE=$HOME/bazeltest_install_base"
- "--test_env=TEST_REPOSITORY_HOME=C:/b/bazeltest_external"
- "--noremote_accept_cached"
- "--config=ci-windows"
build_targets:
- "//src:bazel.exe"
- "//src:bazel_jdk_minimal"
- "//src:test_repos"
- "//src/main/java/..."
- "@test_repos//:srcs"
test_flags:
- "--copt=-w"
- "--host_copt=-w"
- "--test_tag_filters=-no_windows,-slow"
- "--test_env=JAVA_HOME"
- "--test_env=BAZEL_VC"
- "--test_env=TEST_INSTALL_BASE=$HOME/bazeltest_install_base"
- "--test_env=TEST_REPOSITORY_HOME=C:/b/bazeltest_external"
- "--config=ci-windows"
test_targets:
- "//src:embedded_tools_size_test"
- "//src/main/starlark/tests/builtins_bzl/..."
- "//src/test/cpp/..."
- "//src/test/java/com/google/devtools/build/android/..."
- "//src/test/java/com/google/devtools/build/lib/..."
Expand Down Expand Up @@ -381,18 +338,19 @@ tasks:
- test
rbe_ubuntu1804:
platform: ubuntu1804
name: "RBE"
shell_commands:
- sed -i.bak
-e 's/^# android_sdk_repository/android_sdk_repository/'
-e 's/^# android_ndk_repository/android_ndk_repository/' WORKSPACE
- rm -f WORKSPACE.bak
-e 's/^# android_ndk_repository/android_ndk_repository/' WORKSPACE.bzlmod
- rm -f WORKSPACE.bzlmod.bak
build_flags:
- "--config=ubuntu1804_java11"
- "--remote_executor=grpcs://remotebuildexecution.googleapis.com"
- "--jobs=200"
- "--experimental_remote_download_outputs=minimal"
- "--experimental_inmemory_jdeps_files"
- "--experimental_inmemory_dotd_files"
- "--experimental_remote_cache_async"
- "--experimental_remote_merkle_tree_cache"
- "--remote_download_minimal"
build_targets:
- "//src:bazel"
- "//src:bazel_jdk_minimal"
Expand All @@ -401,20 +359,22 @@ tasks:
- "--config=ubuntu1804_java11"
- "--remote_executor=grpcs://remotebuildexecution.googleapis.com"
- "--jobs=200"
- "--experimental_remote_download_outputs=minimal"
- "--experimental_inmemory_jdeps_files"
- "--experimental_inmemory_dotd_files"
- "--experimental_remote_cache_async"
- "--experimental_remote_merkle_tree_cache"
- "--remote_download_minimal"
- "--test_tag_filters=-no_1804"
test_targets:
- "//scripts/..."
- "//src/java_tools/..."
- "//src/main/starlark/tests/builtins_bzl/..."
- "//src/test/..."
- "//src/tools/execlog/..."
- "//src/tools/singlejar/..."
- "//src/tools/workspacelog/..."
- "//third_party/ijar/..."
- "//tools/aquery_differ/..."
- "//tools/python/..."
- "//tools/bash/..."
- "//tools/android/..."
# See https://github.com/bazelbuild/bazel/issues/8033
- "-//src/tools/singlejar:output_jar_simple_test"
Expand Down Expand Up @@ -445,8 +405,8 @@ tasks:
kythe_ubuntu2004:
shell_commands:
- sed -i.bak -e 's/^# android_sdk_repository/android_sdk_repository/'
-e 's/^# android_ndk_repository/android_ndk_repository/' WORKSPACE
- rm -f WORKSPACE.bak
-e 's/^# android_ndk_repository/android_ndk_repository/' WORKSPACE.bzlmod
- rm -f WORKSPACE.bzlmod.bak
index_flags:
- "--define=kythe_corpus=github.com/bazelbuild/bazel"
index_targets_query: "kind(\"cc_(binary|library|test|proto_library) rule\", ...) union kind(\"java_(binary|import|library|plugin|test|proto_library) rule\", ...) union kind(\"proto_library rule\", ...)"
Expand Down
Loading
Loading