Skip to content

Commit

Permalink
Merge bitcoin/bitcoin#30454: build: Introduce CMake-based build system
Browse files Browse the repository at this point in the history 
4105129 cmake: Ignore build subdirectories within source directory (Hennadii Stepanov)
6ce50fd doc: Update for CMake-based build system (Hennadii Stepanov)
9730288 ci: Migrate CI scripts to CMake (Hennadii Stepanov)
c360837 cmake, lint: Adjust `lint_includes_build_config` (Hennadii Stepanov)
3885441 cmake: Add presets for native Windows builds (Hennadii Stepanov)
7681746 cmake: Add vcpkg manifest file (Hennadii Stepanov)
8b6f1c4 cmake: Add `Coverage` and `CoverageFuzz` scripts (Hennadii Stepanov)
65bdbc1 cmake: Add `docs` build target (Hennadii Stepanov)
fb75ebb cmake: Add compiler diagnostic flags (Hennadii Stepanov)
e821f0a cmake: Migrate Guix build scripts to CMake (Hennadii Stepanov)
747adb6 cmake: Add `Maintenance` module (Hennadii Stepanov)
1f60b30 cmake: Add `APPEND_{CPP,C,CXX,LD}FLAGS` cache variables (Hennadii Stepanov)
2b43c45 cmake: Add `AddWindowsResources` module (Hennadii Stepanov)
973a3b0 cmake: Implement `install` build target (Hennadii Stepanov)
84ac35c cmake: Add cross-compiling support (Hennadii Stepanov)
0d01c22 build: Generate `toolchain.cmake` in depends (Hennadii Stepanov)
91a7992 depends: Add host-specific `cmake_system_version` variables (Hennadii Stepanov)
9b31209 depends: Rename `cmake_system` -> `cmake_system_name` (Hennadii Stepanov)
4a5208a Revert "build, qt: Do not install *.prl files" (Hennadii Stepanov)
6522af6 depends: Amend handling flags environment variables (Hennadii Stepanov)
90cec4d cmake: Add `MULTIPROCESS` option (Hennadii Stepanov)
bb1a450 cmake: Build `bitcoin-chainstate` executable (Hennadii Stepanov)
aed38ea cmake: Build `bitcoinkernel` library (Hennadii Stepanov)
975d673 cmake: Build `test_bitcoin-qt` executable (Hennadii Stepanov)
10fcc66 cmake: Add `WITH_DBUS` option (Hennadii Stepanov)
5bb5a4b cmake: Add `libqrencode` optional package support (Hennadii Stepanov)
57a6e2e cmake: Build `bitcoin-qt` executable (Hennadii Stepanov)
30f6429 cmake: Add `WERROR` option (Hennadii Stepanov)
c98d4a4 cmake: Add `REDUCE_EXPORTS` option (Hennadii Stepanov)
a01cb6e cmake: Add `HARDENING` option (Hennadii Stepanov)
a8a2e36 cmake: Add Python-based tests (Hennadii Stepanov)
3d85379 cmake: Add fuzzing options (Hennadii Stepanov)
908530e cmake: Add `SANITIZERS` option (Hennadii Stepanov)
8bb0e85 cmake: Build `bench_bitcoin` executable (Hennadii Stepanov)
8017351 cmake: Add external signer support (Hennadii Stepanov)
353e0c9 cmake: Add `systemtap-sdt` optional package support (Hennadii Stepanov)
d2fda82 cmake: Add `libzmq` optional package support (Hennadii Stepanov)
ae7b39a cmake: Add `libminiupnpc` optional package support (Hennadii Stepanov)
6480e1d cmake: Add `libnatpmp` optional package support (Hennadii Stepanov)
e73e930 cmake: Build `bitcoin-util` executable (Hennadii Stepanov)
027c6d7 cmake: Build `bitcoin-tx` executable (Hennadii Stepanov)
d10c5c3 cmake: Add wallet functionality (Hennadii Stepanov)
ab2e99b cmake: Create test suite for `ctest` (Hennadii Stepanov)
959370b cmake: Build `test_bitcoin` executable (Hennadii Stepanov)
b27bf97 cmake: Build `bitcoin-cli` executable (Hennadii Stepanov)
a9813df cmake: Build `bitcoind` executable (Hennadii Stepanov)
97829ce cmake: Add `FindLibevent` module (Hennadii Stepanov)
3118e40 cmake: Build `bitcoin_consensus` library (Hennadii Stepanov)
809a2f1 cmake: Build `bitcoin_util` static library (Hennadii Stepanov)
0a9a521 cmake: Build `bitcoin_crypto` library (Hennadii Stepanov)
958971f cmake: Build `univalue` static library (Hennadii Stepanov)
752747f cmake: Generate `obj/build.h` header (Hennadii Stepanov)
1f0a78e cmake: Build `minisketch` static library (Hennadii Stepanov)
12bfbc8 cmake: Build `leveldb` static library (Hennadii Stepanov)
51985c5 cmake: Build `crc32c` static library (Hennadii Stepanov)
db7a198 cmake: Build `secp256k1` subtree (Hennadii Stepanov)
dbb7ed1 cmake: Add `ccache` support (Hennadii Stepanov)
cedfdf6 cmake: Redefine/adjust per-configuration flags (Hennadii Stepanov)
b6b5e73 cmake: Add global compiler and linker flags (Hennadii Stepanov)
f983279 cmake: Add `TryAppendLinkerFlag` module (Hennadii Stepanov)
4a0af29 cmake: Add `TryAppendCXXFlags` module (Hennadii Stepanov)
35cffc4 cmake: Add POSIX threads support (Hennadii Stepanov)
fd72d00 cmake: Add position independent code support (Hennadii Stepanov)
07069e2 cmake: Add introspection module (Hennadii Stepanov)
27d687f cmake: Add `config/bitcoin-config.h` support (Hennadii Stepanov)
fe5cdac cmake: Print compiler and linker flags in summary (Hennadii Stepanov)
7068388 cmake: Introduce interface libraries to encapsulate common flags (Hennadii Stepanov)
a2317e2 cmake: Add root `CMakeLists.txt` file (Hennadii Stepanov)

Pull request description:

  This PR introduces a new CMake-based build system, which is a drop-in replacement for the current Autotools-based build system.

  ML announcement: https://groups.google.com/g/bitcoindev/c/hgKkfQWzrTo

  As discussed during the recent CoreDev meetup in April, the switch from Autotools to CMake is intended to happen as soon as possible after branching 28.x off, which means that 29.0 will be built using CMake.

  This PR branch is essentially the [staging branch](https://github.com/hebasto/bitcoin/tree/cmake-staging), with every change reviewed and tested by a group of contributors, including (in alphabetical order):
  - [**achow101**](https://github.com/achow101)
  - [**fanquake**](https://github.com/fanquake)
  - [**maflcko**](https://github.com/maflcko)
  - [**m3dwards**](https://github.com/m3dwards)
  - [**pablomartin4btc**](https://github.com/pablomartin4btc)
  - [**real-or-random**](https://github.com/real-or-random)
  - [**ryanofsky**](https://github.com/ryanofsky)
  - [**sipsorcery**](https://github.com/sipsorcery)
  - [**TheCharlatan**](https://github.com/TheCharlatan)
  - [**theStack**](https://github.com/theStack)
  - [**theuni**](https://github.com/theuni)
  - [**vasild**](https://github.com/vasild)

  Reviewing in a separate staging repo was suggested in bitcoin/bitcoin#27060 (comment).

  The accompanying changes to the OSS-Fuzz project are available in hebasto/oss-fuzz#8.

  Please refer to the [build options parity table](https://gist.github.com/hebasto/2ef97d3a726bfce08ded9df07f7dab5e). The "auto" value is no longer available; non-default values must be specified explicitly. Additionally, the new default values have been chosen to suit the everyday build experience for the majority of developers.

  System requirements for using the CMake-based build system:
  - CMake >= 3.22 (if not available in your system's repository, it can be downloaded from https://cmake.org/download/)
  - a build tool of your choice:
  - any Make (GNU Make is no longer a requirement); GNU Make is still required to build depends
  - Ninja (https://ninja-build.org/)
  - MSBuild
  - Xcode

  A note for Windows users: The default installation of the latest version of MSVC 17.10.4 includes both CMake 3.28.3 and the vcpkg package manager).

  ---

  We, the build system developers, kindly ask reviewers to refrain from making suggestions that are not directly related to the migration process or can be implemented separately. Bugs in the scripts and errors in the updated documentation should be the focus of this PR. Please be advised that comments not aligned with this PR's goal may be ignored.

  Thank you all for your understanding.

ACKs for top commit:
  maflcko:
    review ACK 4105129 🐥
  sipsorcery:
    ACK 4105129.
  vasild:
    ACK 4105129
  TheCharlatan:
    ACK 4105129
  pablomartin4btc:
    tACK 4105129
  i-am-yuvi:
    tACK [`4105129`](bitcoin/bitcoin@4105129)
  theuni:
    ACK 4105129.
  fanquake:
    ACK 4105129

Tree-SHA512: 6c1445054436c6c00ad63bfa0f19d64091a2b25c9bd694f85bf2218ac358ffb774d6c000685b3ca1e9b50401babed989fa2a0694b774c211d226bfd1944c9b39
  • Loading branch information
@fanquake
fanquake committed Aug 28, 2024
2 parents 80f00ca + 4105129 commit 338bc2c
Show file tree
Hide file tree
Showing 113 changed files with 5,872 additions and 518 deletions.
149 changes: 34 additions & 115 deletions .github/workflows/ci.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,12 +67,12 @@ jobs:
echo "TEST_BASE=$(git rev-list -n$((${{ env.MAX_COUNT }} + 1)) --reverse HEAD $EXCLUDE_MERGE_BASE_ANCESTORS | head -1)" >> "$GITHUB_ENV"
- run: |
sudo apt-get update
sudo apt-get install clang ccache build-essential libtool autotools-dev automake pkg-config bsdmainutils python3-zmq libevent-dev libboost-dev libsqlite3-dev libdb++-dev systemtap-sdt-dev libminiupnpc-dev libnatpmp-dev qtbase5-dev qttools5-dev qttools5-dev-tools qtwayland5 libqrencode-dev -y
sudo apt-get install clang ccache build-essential cmake pkg-config bsdmainutils python3-zmq libevent-dev libboost-dev libsqlite3-dev libdb++-dev systemtap-sdt-dev libminiupnpc-dev libnatpmp-dev qtbase5-dev qttools5-dev qttools5-dev-tools qtwayland5 libqrencode-dev -y
- name: Compile and run tests
run: |
# Run tests on commits after the last merge commit and before the PR head commit
# Use clang++, because it is a bit faster and uses less memory than g++
git rebase --exec "echo Running test-one-commit on \$( git log -1 ) && ./autogen.sh && CC=clang CXX=clang++ ./configure --with-incompatible-bdb && make clean && make -j $(nproc) check && ./test/functional/test_runner.py -j $(( $(nproc) * 2 ))" ${{ env.TEST_BASE }}
git rebase --exec "echo Running test-one-commit on \$( git log -1 ) && CC=clang CXX=clang++ cmake -B build -DBUILD_GUI=ON -DBUILD_BENCH=ON -DBUILD_FUZZ_BINARY=ON -DWITH_BDB=ON -DWITH_NATPMP=ON -DWITH_MINIUPNPC=ON -DWITH_USDT=ON && cmake --build build -j $(nproc) && ctest --test-dir build -j $(nproc) && ./build/test/functional/test_runner.py -j $(( $(nproc) * 2 ))" ${{ env.TEST_BASE }}
macos-native-x86_64:
name: 'macOS 13 native, x86_64, no depends, sqlite only, gui'
Expand Down Expand Up @@ -139,18 +139,17 @@ jobs:
if: github.repository != 'bitcoin-core/gui' || github.event_name == 'pull_request'

env:
CCACHE_MAXSIZE: '200M'
CI_CCACHE_VERSION: '4.7.5'
CI_QT_CONF: '-release -silent -opensource -confirm-license -opengl desktop -static -static-runtime -mp -qt-zlib -qt-pcre -qt-libpng -nomake examples -nomake tests -nomake tools -no-angle -no-dbus -no-gif -no-gtk -no-ico -no-icu -no-libjpeg -no-libudev -no-sql-sqlite -no-sql-odbc -no-sqlite -no-vulkan -skip qt3d -skip qtactiveqt -skip qtandroidextras -skip qtcharts -skip qtconnectivity -skip qtdatavis3d -skip qtdeclarative -skip doc -skip qtdoc -skip qtgamepad -skip qtgraphicaleffects -skip qtimageformats -skip qtlocation -skip qtlottie -skip qtmacextras -skip qtmultimedia -skip qtnetworkauth -skip qtpurchasing -skip qtquick3d -skip qtquickcontrols -skip qtquickcontrols2 -skip qtquicktimeline -skip qtremoteobjects -skip qtscript -skip qtscxml -skip qtsensors -skip qtserialbus -skip qtserialport -skip qtspeech -skip qtsvg -skip qtvirtualkeyboard -skip qtwayland -skip qtwebchannel -skip qtwebengine -skip qtwebglplugin -skip qtwebsockets -skip qtwebview -skip qtx11extras -skip qtxmlpatterns -no-openssl -no-feature-bearermanagement -no-feature-printdialog -no-feature-printer -no-feature-printpreviewdialog -no-feature-printpreviewwidget -no-feature-sql -no-feature-sqlmodel -no-feature-textbrowser -no-feature-textmarkdownwriter -no-feature-textodfwriter -no-feature-xml'
CI_QT_DIR: 'qt-everywhere-src-5.15.11'
CI_QT_URL: 'https://download.qt.io/official_releases/qt/5.15/5.15.11/single/qt-everywhere-opensource-src-5.15.11.zip'
PYTHONUTF8: 1
TEST_RUNNER_TIMEOUT_FACTOR: 40

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Remove non-MSVC tool installations
run: |
Remove-Item -Path "$env:ProgramFiles/CMake" -Recurse -Force
- name: Configure Developer Command Prompt for Microsoft Visual C++
# Using microsoft/setup-msbuild is not enough.
uses: ilammy/msvc-dev-cmd@v1
Expand All @@ -159,141 +158,60 @@ jobs:

- name: Get tool information
run: |
cmake -version | Tee-Object -FilePath "cmake_version"
Write-Output "---"
msbuild -version | Tee-Object -FilePath "msbuild_version"
$env:VCToolsVersion | Tee-Object -FilePath "toolset_version"
$env:CI_QT_URL | Out-File -FilePath "qt_url"
$env:CI_QT_CONF | Out-File -FilePath "qt_conf"
py -3 --version
Write-Host "PowerShell version $($PSVersionTable.PSVersion.ToString())"
- name: Restore static Qt cache
id: static-qt-cache
uses: actions/cache/restore@v4
with:
path: C:\Qt_static
key: ${{ github.job }}-static-qt-${{ hashFiles('msbuild_version', 'qt_url', 'qt_conf') }}

- name: Build static Qt. Download
if: steps.static-qt-cache.outputs.cache-hit != 'true'
shell: cmd
run: |
curl --location --output C:\qt-src.zip %CI_QT_URL%
choco install --yes --no-progress jom
- name: Build static Qt. Expand source archive
if: steps.static-qt-cache.outputs.cache-hit != 'true'
shell: cmd
run: tar -xf C:\qt-src.zip -C C:\

- name: Build static Qt. Create build directory
if: steps.static-qt-cache.outputs.cache-hit != 'true'
run: |
Rename-Item -Path "C:\$env:CI_QT_DIR" -NewName "C:\qt-src"
New-Item -ItemType Directory -Path "C:\qt-src\build"
- name: Build static Qt. Configure
if: steps.static-qt-cache.outputs.cache-hit != 'true'
working-directory: C:\qt-src\build
shell: cmd
run: ..\configure %CI_QT_CONF% -prefix C:\Qt_static

- name: Build static Qt. Build
if: steps.static-qt-cache.outputs.cache-hit != 'true'
working-directory: C:\qt-src\build
shell: cmd
run: jom

- name: Build static Qt. Install
if: steps.static-qt-cache.outputs.cache-hit != 'true'
working-directory: C:\qt-src\build
shell: cmd
run: jom install

- name: Save static Qt cache
if: steps.static-qt-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@v4
with:
path: C:\Qt_static
key: ${{ github.job }}-static-qt-${{ hashFiles('msbuild_version', 'qt_url', 'qt_conf') }}

- name: Ccache installation cache
id: ccache-installation-cache
uses: actions/cache@v4
with:
path: |
C:\ProgramData\chocolatey\lib\ccache
C:\ProgramData\chocolatey\bin\ccache.exe
C:\ccache\cl.exe
key: ${{ github.job }}-ccache-installation-${{ env.CI_CCACHE_VERSION }}

- name: Install Ccache
if: steps.ccache-installation-cache.outputs.cache-hit != 'true'
run: |
choco install --yes --no-progress ccache --version=$env:CI_CCACHE_VERSION
New-Item -ItemType Directory -Path "C:\ccache"
Copy-Item -Path "$env:ChocolateyInstall\lib\ccache\tools\ccache-$env:CI_CCACHE_VERSION-windows-x86_64\ccache.exe" -Destination "C:\ccache\cl.exe"
- name: Restore Ccache cache
id: ccache-cache
uses: actions/cache/restore@v4
with:
path: ~/AppData/Local/ccache
key: ${{ github.job }}-ccache-${{ github.run_id }}
restore-keys: ${{ github.job }}-ccache-

- name: Using vcpkg with MSBuild
run: |
Set-Location "$env:VCPKG_INSTALLATION_ROOT"
Add-Content -Path "triplets\x64-windows.cmake" -Value "set(VCPKG_BUILD_TYPE release)"
Add-Content -Path "triplets\x64-windows-static.cmake" -Value "set(VCPKG_BUILD_TYPE release)"
.\vcpkg.exe --vcpkg-root "$env:VCPKG_INSTALLATION_ROOT" integrate install
git rev-parse HEAD | Tee-Object -FilePath "$env:GITHUB_WORKSPACE\vcpkg_commit"
- name: vcpkg tools cache
uses: actions/cache@v4
with:
path: C:/vcpkg/downloads/tools
key: ${{ github.job }}-vcpkg-tools

- name: vcpkg binary cache
uses: actions/cache@v4
- name: Restore vcpkg binary cache
uses: actions/cache/restore@v4
id: vcpkg-binary-cache
with:
path: ~/AppData/Local/vcpkg/archives
key: ${{ github.job }}-vcpkg-binary-${{ hashFiles('vcpkg_commit', 'msbuild_version', 'toolset_version', 'build_msvc/vcpkg.json') }}

- name: Generate project files
run: py -3 build_msvc\msvc-autogen.py
key: ${{ github.job }}-vcpkg-binary-${{ hashFiles('cmake_version', 'msbuild_version', 'toolset_version', 'vcpkg.json') }}

- name: Build
shell: cmd
- name: Generate build system
run: |
ccache --zero-stats
msbuild build_msvc\bitcoin.sln -property:CLToolPath=C:\ccache;CLToolExe=cl.exe;UseMultiToolTask=true;Configuration=Release -maxCpuCount -verbosity:minimal -noLogo
- name: Ccache stats
run: ccache --show-stats
cmake -B build --preset vs2022-static -DCMAKE_TOOLCHAIN_FILE="$env:VCPKG_INSTALLATION_ROOT\scripts\buildsystems\vcpkg.cmake" -DBUILD_BENCH=ON -DBUILD_FUZZ_BINARY=ON -DWERROR=ON
- name: Save Ccache cache
- name: Save vcpkg binary cache
uses: actions/cache/save@v4
if: github.event_name != 'pull_request' && steps.ccache-cache.outputs.cache-hit != 'true'
if: github.event_name != 'pull_request' && steps.vcpkg-binary-cache.outputs.cache-hit != 'true'
with:
path: ~/AppData/Local/ccache
# https://github.com/actions/cache/blob/main/tips-and-workarounds.md#update-a-cache
key: ${{ github.job }}-ccache-${{ github.run_id }}

- name: Run unit tests
run: src\test_bitcoin.exe -l test_suite

- name: Run benchmarks
run: src\bench_bitcoin.exe -sanity-check
path: ~/AppData/Local/vcpkg/archives
key: ${{ github.job }}-vcpkg-binary-${{ hashFiles('cmake_version', 'msbuild_version', 'toolset_version', 'vcpkg.json') }}

- name: Run util tests
run: py -3 test\util\test_runner.py
- name: Build
working-directory: build
run: |
cmake --build . -j $env:NUMBER_OF_PROCESSORS --config Release
- name: Run rpcauth test
run: py -3 test\util\rpcauth-test.py
- name: Run test suite
working-directory: build
run: |
ctest -j $env:NUMBER_OF_PROCESSORS -C Release
- name: Run functional tests
working-directory: build
env:
BITCOIND: '${{ github.workspace }}\build\src\Release\bitcoind.exe'
BITCOINCLI: '${{ github.workspace }}\build\src\Release\bitcoin-cli.exe'
BITCOINUTIL: '${{ github.workspace }}\build\src\Release\bitcoin-util.exe'
BITCOINWALLET: '${{ github.workspace }}\build\src\Release\bitcoin-wallet.exe'
TEST_RUNNER_EXTRA: ${{ github.event_name != 'pull_request' && '--extended' || '' }}
shell: cmd
run: py -3 test\functional\test_runner.py --jobs %NUMBER_OF_PROCESSORS% --ci --quiet --tmpdirprefix=%RUNNER_TEMP% --combinedlogslen=99999999 --timeout-factor=%TEST_RUNNER_TIMEOUT_FACTOR% %TEST_RUNNER_EXTRA%
Expand All @@ -306,8 +224,9 @@ jobs:
git log -1
- name: Run fuzz binaries
working-directory: build
env:
BITCOINFUZZ: "${{ github.workspace}}\\src\\fuzz.exe"
BITCOINFUZZ: '${{ github.workspace }}\build\src\test\fuzz\Release\fuzz.exe'
shell: cmd
run: py -3 test\fuzz\test_runner.py --par %NUMBER_OF_PROCESSORS% --loglevel DEBUG %RUNNER_TEMP%\qa-assets\fuzz_seed_corpus

Expand Down
5 changes: 5 additions & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# Build subdirectories.
/*build*
!/build-aux
!/build_msvc

*.tar.gz

*.exe
Expand Down
Loading

0 comments on commit 338bc2c

Please sign in to comment.