Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: bazel foundation #19946

Draft
wants to merge 117 commits into
base: dev
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 98 commits
Commits
Show all changes
117 commits
Select commit Hold shift + click to select a range
7eebf73
bytes: reference header in current directory
dotnwat Jun 12, 2024
281b8c9
bazel: add third party dependencies
dotnwat Jun 6, 2024
9895d33
bazel: build wrapper
dotnwat Jun 9, 2024
39d3d9a
bazel: add baes library bazel build
dotnwat Jun 8, 2024
61a0fe1
bazel: add ssx library bazel build
dotnwat Jun 8, 2024
65ffd83
bazel: add strings library bazel build
dotnwat Jun 8, 2024
80802ae
bazel: add bytes library bazel build
dotnwat Jun 8, 2024
e5aa44f
bazel: add container library bazel build
dotnwat Jun 8, 2024
06ff960
bazel: add utils library bazel build
dotnwat Jun 8, 2024
b90e0ca
bazel: add crypto library bazel build
dotnwat Jun 8, 2024
2b47fbd
bazel: add random library bazel build
dotnwat Jun 8, 2024
b8d3ed2
bazel: add serde library bazel build
dotnwat Jun 8, 2024
e8efde5
bazel: add test_utils library bazel build
dotnwat Jun 8, 2024
6826627
bazel: update seastar and add config options
dotnwat Jun 19, 2024
e2fcf0a
bazel: add missing hwloc depdendency
dotnwat Jun 19, 2024
be287e1
bazel: use label conditions
dotnwat Jun 19, 2024
e58bd47
bazel: use inherited c++ standard setting
dotnwat Jun 19, 2024
f9319a1
bazel: clarify seastar build comments
dotnwat Jun 19, 2024
82e7954
bazel: make strings library fine grained
dotnwat Jun 19, 2024
4e96dcc
bazel: add no-seastar boost test macro
dotnwat Jun 19, 2024
bbfde7d
bazel: add strings library tests
dotnwat Jun 19, 2024
9a1f8e6
bazel: add crypto tests
dotnwat Jun 19, 2024
b52e2a7
bazel: make random library more fine grained
dotnwat Jun 20, 2024
169f910
bazel: add random library tests
dotnwat Jun 20, 2024
ddd4cbd
bazel: allow defines to pass through for test macro
dotnwat Jun 20, 2024
cc035b8
bazel: add tests to utils library
dotnwat Jun 20, 2024
01abe7a
bazel: add base outcome test
dotnwat Jun 20, 2024
0bbd40e
Merge remote-tracking branch 'origin/dev' into bazel
dotnwat Jun 20, 2024
075f5eb
bazel: use a better name for seastar numactl option
dotnwat Jun 21, 2024
1fcd770
bazel: add ssx tests
dotnwat Jun 21, 2024
fad3b00
bazel: split build from test helpers
dotnwat Jun 21, 2024
d698141
Merge remote-tracking branch 'origin/dev' into bazel
dotnwat Jun 21, 2024
329d13a
bazel: update skylib
dotnwat Jun 21, 2024
590810f
bazel: upgrade to bazel 7.2
dotnwat Jun 21, 2024
a89af08
bazel: fix dependency reference
dotnwat Jun 21, 2024
ca22e2b
bazel: remove unused import
dotnwat Jun 22, 2024
2887843
bazel: pass definitions through library macro
dotnwat Jun 22, 2024
f5da85d
bazel: add named_type to bazel build
dotnwat Jun 22, 2024
f2ec24e
bazel: add hashing library
dotnwat Jun 22, 2024
b5d51b7
bazel: add utils object_pool
dotnwat Jun 22, 2024
ed9ba45
bazel: upgrade rules_proto
dotnwat Jun 23, 2024
ad42e34
bazel: add rpc compiler macro
dotnwat Jun 23, 2024
ee8e91d
bazel: generate raft rpc service header
dotnwat Jun 23, 2024
ecc8b4d
bazel: generate cluster rpc headers
dotnwat Jun 23, 2024
5819f3c
bazel: generate transform rpc headers
dotnwat Jun 23, 2024
5563372
bazel: use python rules from rules_python
dotnwat Jun 23, 2024
b4e9c43
bazel: remove empty dependency list
dotnwat Jun 23, 2024
4080858
bazel: generate seastar swagger from json spec
dotnwat Jun 23, 2024
fa29de0
bazel: add buildifier tool for bazel file formatting
dotnwat Jun 23, 2024
8007cc2
bazel: split fips build from rest of openssl build
dotnwat Jun 23, 2024
745a230
bazel: use explicit flags instead of compilation_mode
dotnwat Jun 25, 2024
2f255c9
Merge remote-tracking branch 'origin/dev' into bazel
dotnwat Jun 25, 2024
0f7fd32
bazel: keep llvm option parsing on same line
dotnwat Jun 26, 2024
5305841
bazel: add thin lto to release flags
dotnwat Jun 26, 2024
a7e178f
bazel: use openssl flags in the openssl-fips build
dotnwat Jun 26, 2024
9bc2053
bazel: avoid leaking avro include path to dependents
dotnwat Jun 26, 2024
fed5d40
bazel: upgrade seastar
dotnwat Jun 26, 2024
b813457
bazel: always build seastar with openssl support
dotnwat Jun 26, 2024
d61347e
Merge remote-tracking branch 'origin/dev' into bazel
dotnwat Jun 26, 2024
22bc0c0
bazel: remove unused seastar dependencies
dotnwat Jun 26, 2024
538f55a
bazel: add module and function docstrings
dotnwat Jun 27, 2024
21b9683
bazel: fix unused variable warning
dotnwat Jun 27, 2024
53a08f3
bazel: remove use of globbing in serde
dotnwat Jun 27, 2024
e41c776
bazel: add buildifier tool
dotnwat Jun 27, 2024
52e690b
tools: add tools/bazel wrapper
dotnwat Jun 27, 2024
f193188
Merge remote-tracking branch 'origin/dev' into bazel
dotnwat Jun 27, 2024
ea828bc
bazel: allow rules_python to run with root user
dotnwat Jun 28, 2024
baa87be
bazel: add dependency for libpciaccess
dotnwat Jun 28, 2024
4302062
bazel: fix formatting
dotnwat Jun 28, 2024
f9678d2
bazel: add unresolved address utility
dotnwat Jun 28, 2024
dbc06de
bazel: add json library
dotnwat Jun 28, 2024
7cefc24
Merge remote-tracking branch 'origin/dev' into bazel
dotnwat Jun 28, 2024
83c1c21
bazel: add fix variant of buildifier command
dotnwat Jun 29, 2024
c2db0a8
bazel: add utils:uuid library
dotnwat Jun 29, 2024
69110a4
bazel: add reflection utility libraries
dotnwat Jun 29, 2024
a79fb21
bazel: add additional serde tag_invoke helpers
dotnwat Jun 29, 2024
8912807
bazel: add utils::uuid test
dotnwat Jun 29, 2024
97a9e9a
bazel: add utils:xid library and test
dotnwat Jun 29, 2024
5dcaaf5
Merge remote-tracking branch 'origin/dev' into HEAD
dotnwat Jun 29, 2024
a9b8a88
bazel: ignore build directories
dotnwat Jun 29, 2024
29982fe
bazel: add missing header to seastar build
dotnwat Jun 29, 2024
ec7a722
bazel: allow local_defines pass through to library macro
dotnwat Jun 30, 2024
1f5145b
bazel: add utils:static_deleter_fn library
dotnwat Jun 30, 2024
956faf7
bazel: add utils:functional library
dotnwat Jun 30, 2024
f5a2039
bazel: add ssx:future-util library
dotnwat Jun 30, 2024
8958c57
bazel: add reflection:adl library
dotnwat Jun 30, 2024
f1c0574
bazel: add several serde rw tag_invoke libraries
dotnwat Jun 30, 2024
0e3c42d
bazel: add utils:to_string library
dotnwat Jun 30, 2024
4505f93
bazel: add model library
dotnwat Jun 30, 2024
d5e38a1
bazel: add compression library
dotnwat Jun 30, 2024
20ec04c
Merge remote-tracking branch 'origin/dev' into bazel
dotnwat Jun 30, 2024
00e29d5
bazel: enable layering_check by default
dotnwat Jun 30, 2024
8b74a72
Merge pull request #17 from dotnwat/layering-check
dotnwat Jun 30, 2024
f862f7b
bazel: remove non-library code from avro
dotnwat Jun 30, 2024
b813371
bazel: add missing avro dependencies
dotnwat Jun 30, 2024
e6d916a
bazel: add missing utils:uuid dependency
dotnwat Jun 30, 2024
8484270
bazel: build xxhash explicitly
dotnwat Jun 30, 2024
7985abd
bazel: add missing system dependencies
dotnwat Jun 30, 2024
52d8d53
bazel/thirdparty: added debug config flags to seastar
mmaslankaprv Jul 2, 2024
be81892
build/bazelrc: use default system allocator when using sanitizers
mmaslankaprv Jul 2, 2024
1554594
Merge remote-tracking branch 'origin/dev' into bazel
dotnwat Jul 2, 2024
eeeda55
bazel: enable warnings for redpanda targets
dotnwat Jul 1, 2024
aecf8d1
bazel: use custom build for libpciaccess
dotnwat Jul 2, 2024
dbadf9b
bazel: add keep_going flag to build
dotnwat Jul 2, 2024
7304569
Merge remote-tracking branch 'michal/seastar-debug-opts' into bazel
dotnwat Jul 2, 2024
107ed92
bazel: remove whitespace
dotnwat Jul 2, 2024
95a8172
Merge pull request #20 from dotnwat/pciaccess-warnings
dotnwat Jul 2, 2024
023445c
bazel: add utils:inet_address library
dotnwat Jul 2, 2024
ff8a6ae
bazel: add schema_registry:config library
dotnwat Jul 2, 2024
65e2f6d
bazel: add storage:config library
dotnwat Jul 2, 2024
20be636
bazel: add serde:chrono library
dotnwat Jul 2, 2024
68c5582
Merge remote-tracking branch 'origin/dev' into bazel-config
dotnwat Jul 3, 2024
2f5dc42
bazel: add serde:bool_class library
dotnwat Jul 3, 2024
c2d71fb
bazel: add missing serde deps to model library
dotnwat Jul 3, 2024
5393109
bazel: remove rw/vector from generic serde library
dotnwat Jul 3, 2024
4281447
bazel: add security:config library
dotnwat Jul 2, 2024
c83f8f4
bazel: add config library
dotnwat Jul 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .bazelignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
build
vbuild
65 changes: 65 additions & 0 deletions .bazelrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# use a compiler name that doesn't symlink to ccache
build --action_env=CC=clang-16 --host_action_env=CC=clang-16
build --action_env=CXX=clang++-16 --host_action_env=CXX=clang++-16
build --action_env=BAZEL_COMPILER=clang
build --cxxopt=-std=c++20 --host_cxxopt=-std=c++20
build --linkopt -fuse-ld=lld

# https://github.com/bazelbuild/rules_foreign_cc/issues/1065
# https://github.com/bazelbuild/rules_foreign_cc/issues/1186#issuecomment-2053550487
build --host_action_env=CXXFLAGS=-Wno-int-conversion
build --action_env=CXXFLAGS=-Wno-int-conversion
build --host_action_env=CFLAGS=-Wno-int-conversion
build --action_env=CFLAGS=-Wno-int-conversion

common --registry=https://bcr.bazel.build

# =================================
# Sanitizer
# =================================
build:sanitizer --copt -fsanitize=address,undefined,vptr,function
build:sanitizer --linkopt -fsanitize=address,undefined,vptr,function
build:sanitizer --linkopt --rtlib=compiler-rt
build:sanitizer --linkopt -fsanitize-link-c++-runtime
build:sanitizer --copt -O1

# =================================
# Security
# =================================
build:cfi --copt -fsanitize=cfi
build:cfi --copt -fvisibility=hidden
build:cfi --copt -flto

# fortify source requires a high optimization level. when using this feature add
# `--copt -O2` or combine with a compilation mode like `-c opt`. the default
# bazel toolchain adds _FORTIFY_SOURCE=1 in `-c opt`, so first clear the
# definition to avoid -Wmacro-redefined warnings.
build:fortify-source --copt -U_FORTIFY_SOURCE
build:fortify-source --copt -D_FORTIFY_SOURCE=2

build:safe-stack --copt -fsanitize=safe-stack
build:stack-clash --copt -fstack-clash-protection
build:stack-protector --copt -fstack-protector-strong
build:relro --copt -fno-plt
build:relro --linkopt -Wl,-z,relro,-z,now

# Use `--config=secure` for all supported security settings
build:secure --config=fortify-source
build:secure --config=stack-clash
build:secure --config=stack-protector
build:secure --config=relro
# Cannot enable until all external deps support
#build:secure --config=cfi
#build:secure --config=safe-stack

# =================================
# Release
# =================================
build:release --compilation_mode opt
build:release --config=secure
build:release --copt -mllvm --copt -inline-threshold=2500
dotnwat marked this conversation as resolved.
Show resolved Hide resolved
build:release --linkopt=-flto
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this gets enabled in the current release build (and also I don't think it would do anything)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thin-lto? i added it by request of @rockwotj, i'll let him chime in. good point about it not being used in the current build!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My intent was to preserve this behavior:

set_property(TARGET redpanda PROPERTY INTERPROCEDURAL_OPTIMIZATION ON)

Does that line to something different or am I misunderstanding what that does?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that quoted like from cmake applies to only the final binary linking, and doesn't take affect transitively across the entire tree, but would need to be enabled globally to have an affect. i'm also not sure i know what i'm talking about :P

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was referring to the vtools build yeah. It needs to be enabled for the compile steps as well yeah. I think passing it only at link time does have some effect like it being able to remove unused functions from the final binary but it's not the full thing that people usually expect.

Hadn't seen that bit in the public build. Are we sure that actually evaluates to true? Just had a quick look and it doesn't get passed for the compile steps at least (not sure whether that is what that cmake logic would do or not).


build --@seastar//:openssl=True

try-import %workspace%/user.bazelrc
1 change: 1 addition & 0 deletions .bazelversion
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
7.2.0
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -263,3 +263,6 @@ vbuild/

#asdf tool version file
.tool-versions

# bazel
user.bazelrc
83 changes: 83 additions & 0 deletions MODULE.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
module(
name = "redpanda",
repo_name = "com_github_redpanda_data_redpanda",
)

bazel_dep(name = "hedron_compile_commands", dev_dependency = True)
git_override(
module_name = "hedron_compile_commands",
commit = "a14ad3a64e7bf398ab48105aaa0348e032ac87f8",
remote = "https://github.com/hedronvision/bazel-compile-commands-extractor.git",
)

bazel_dep(name = "rules_boost", repo_name = "com_github_nelhage_rules_boost")
archive_override(
module_name = "rules_boost",
integrity = "sha256-PcGxb6Ffi1CajH1PIqlAJ6mgqQ5/XW3NNL7TLZvlSBs=",
strip_prefix = "rules_boost-64bf4814222a6782fd0e7536532a257d7fdc9d80",
urls = "https://github.com/nelhage/rules_boost/archive/64bf4814222a6782fd0e7536532a257d7fdc9d80.tar.gz",
)

non_module_boost_repositories = use_extension("@com_github_nelhage_rules_boost//:boost/repositories.bzl", "non_module_dependencies")
use_repo(
non_module_boost_repositories,
"boost",
)

bazel_dep(name = "abseil-cpp", version = "20240116.1")
bazel_dep(name = "bazel_skylib", version = "1.7.1")
bazel_dep(name = "crc32c", version = "1.1.0")
bazel_dep(name = "fmt", version = "8.1.1")
bazel_dep(name = "googletest", version = "1.14.0.bcr.1")
bazel_dep(name = "liburing", version = "2.5")
bazel_dep(name = "lz4", version = "1.9.4")
bazel_dep(name = "platforms", version = "0.0.10")
bazel_dep(name = "protobuf", version = "26.0")
bazel_dep(name = "re2", version = "2023-09-01")
bazel_dep(name = "rules_foreign_cc", version = "0.10.1")
bazel_dep(name = "rules_proto", version = "6.0.2")
bazel_dep(name = "yaml-cpp", version = "0.8.0")
bazel_dep(name = "zlib", version = "1.3")
bazel_dep(name = "zstd", version = "1.5.6")

bazel_dep(name = "buildifier_prebuilt", version = "6.4.0", dev_dependency = True)
bazel_dep(name = "rules_python", version = "0.33.2", dev_dependency = True)

python = use_extension("@rules_python//python/extensions:python.bzl", "python", dev_dependency = True)
python.toolchain(
ignore_root_user_error = True,
is_default = True,
python_version = "3.12",
)

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
extra_pip_args = ["--require-hashes"],
hub_name = "python_deps",
python_version = "3.12",
requirements_lock = "//bazel/thirdparty:requirements.txt",
)
use_repo(pip, "python_deps")

non_module_dependencies = use_extension("//bazel:extensions.bzl", "non_module_dependencies")
use_repo(non_module_dependencies, "ada")
use_repo(non_module_dependencies, "avro")
use_repo(non_module_dependencies, "base64")
use_repo(non_module_dependencies, "c-ares")
use_repo(non_module_dependencies, "hdrhistogram")
use_repo(non_module_dependencies, "hwloc")
use_repo(non_module_dependencies, "krb5")
use_repo(non_module_dependencies, "libpciaccess")
use_repo(non_module_dependencies, "libxml2")
use_repo(non_module_dependencies, "lksctp")
use_repo(non_module_dependencies, "numactl")
use_repo(non_module_dependencies, "openssl")
use_repo(non_module_dependencies, "openssl-fips")
use_repo(non_module_dependencies, "rapidjson")
use_repo(non_module_dependencies, "roaring")
use_repo(non_module_dependencies, "seastar")
use_repo(non_module_dependencies, "snappy")
use_repo(non_module_dependencies, "unordered_dense")
use_repo(non_module_dependencies, "wasmtime")
use_repo(non_module_dependencies, "xxhash")
use_repo(non_module_dependencies, "xz")
Loading