external_deps: revamp lib build, add cmake and configure wrapper, update lib feature disablement, update lib versions #1433

illwieckz · 2024-11-11T16:57:00Z

This is WIP, current external_deps build status:

system	deps build	static build	test
linux-amd64-default	✅️	✅️	✅️
linux-arm64-default	✅️
linux-i686-default	✅️	✅️	✅️
linux-armhf-default	✅️
windows-amd64-mingw	✅️	N/A	✅️
windows-i686-mingw	✅️ Buster ❌️ GLEW Noble	N/A	✅️
windows-amd64-msvc	✅️
windows-i686-msvc	✅️
macos-amd64-default	✅️	N/A	✅️

~~I haven't tested if the engine builds and runs properly with those.~~ WIP

What this PR does:

Use CMake when packages provide a CMakeLists.txt file
Factorize much code
Make a nice list of package base url, so it's easy to click all of them and check for new versions
Update package versions
Update package build options

illwieckz · 2024-11-11T17:00:47Z

For windows-i686-mingw I get this with GLEW:

i686-w64-mingw32-ld -nostdlib -shared -soname libglew32.dll --out-implib lib/libglew32.dll.a
     -o lib/glew32.dll tmp/linux-mingw32/default/shared/glew.o
     -Lexternal_deps/build-windows-i686-mingw_10/prefix/lib -L/usr/i686-w64-mingw32/lib
     -lopengl32 -lgdi32 -luser32 -lkernel32 
i686-w64-mingw32-ld: tmp/linux-mingw32/default/shared/glew.o:glew.c:(.text+0x141ab):
 undefined reference to `strlen'

Everything else build.

illwieckz · 2024-11-11T17:23:07Z

For windows-amd64-msvc I get this with Vorbis:

[ 84%] Linking C shared library libvorbis.dll
/usr/bin/x86_64-w64-mingw32-ld:
 external_deps/build-windows-amd64-msvc_10/vorbis/libvorbis-1.3.7/win32/vorbis.def:3: syntax error
/usr/bin/x86_64-w64-mingw32-ld:
 external_deps/build-windows-amd64-msvc_10/vorbis/libvorbis-1.3.7/win32/vorbis.def: file format not recognized; treating as linker script
/usr/bin/x86_64-w64-mingw32-ld:
 external_deps/build-windows-amd64-msvc_10/vorbis/libvorbis-1.3.7/win32/vorbis.def:2: syntax error

Everything else build.

illwieckz · 2024-11-11T18:28:39Z

OGG has a CMakeLists.txt file but no install target, so I returned back to configure for this one.
Opus requires at least CMake 3.16.

illwieckz · 2024-11-11T18:32:24Z

When building newer Opus on Debian buster (the distro we use for our releases), I get this:

opus/opus-1.5.2/silk/x86/NSQ_del_dec_avx2.c:959:43: warning: implicit declaration of function '_mm_loadu_si64'; did you mean '_mm_loadu_si32'? [-Wimplicit-function-declaration]
         __m256i x = _mm256_cvtepi16_epi64(_mm_loadu_si64(&x16[i]));
                                           ^~~~~~~~~~~~~~
                                           _mm_loadu_si32

Debian Buster provides GCC 8, but GCC 11.3 or GCC 12 may be required:
https://stackoverflow.com/questions/72837929/mm-loadu-si32-not-recognized-by-gcc-on-ubuntu

illwieckz · 2024-11-11T18:41:29Z

When building newer Opus on Debian buster (the distro we use for our releases), I get this: […]

Telling Opus to not assume more than SSE2 fixed that.

illwieckz · 2024-11-11T18:54:40Z

So, the MinGW GLEW and MSVC Vorbis errors are the only ones.

illwieckz · 2024-11-11T19:26:25Z

So the Vorbis build error is actually a bug:

I added a workaround.

illwieckz · 2024-11-11T19:31:40Z

I don't get why I get that GLEW build error with MinGW, the build function has not been modified, and the version has not been updated.

illwieckz · 2024-11-11T19:35:07Z

I don't get why I get that GLEW build error with MinGW, the build function has not been modified, and the version has not been updated.

Also the code is the same for both windows-amd64-msvc, windows-i686-msvc, windows-amd64-mingw and windows-i686-mingw, but it only fails with windows-i686-mingw… That doesn't make sense.

illwieckz · 2024-11-11T19:40:17Z

So, I don't know what happened, now I don't reproduce the MinGW GLEW error… Maybe I gorgot to prune the prefix folder and some stray files messed-up…

illwieckz · 2024-11-11T19:41:54Z

Ah, I now see something: I reproduce the bug with MinGW from Ubuntu 24.04 Noble, not with MinGW from Debian 10 Buster. So, since we produce release builds with Debian Buster, it's not a big problem, but it should be fixed for the future…

slipher · 2024-11-11T21:24:14Z

What's the purpose of migrating things to build with CMake?

illwieckz · 2024-11-16T21:31:33Z

So with this a static build for linux-amd64 completes and runs.

slipher · 2024-11-16T21:47:57Z

So with this a static build for linux-amd64 completes and runs.

Huh? It worked before, so I have a hard time believing that is suddenly necessary to change the build system of 8 packages.

illwieckz · 2024-11-16T21:49:58Z

I never said this is a response to “What's the purpose of migrating things to build with CMake?”, I'm just reporting the status of me testing that branch. I said in first post:

I haven't tested if the engine builds and runs properly with those.

So now I'm running those tests.

DolceTriade · 2024-11-16T22:04:08Z

What's the motivation behind this change?

illwieckz · 2024-11-18T01:00:22Z

Purposes:

Update packages to latest versions,
Update packages build options for latest versions (for example curl adds protocols, so we have to disable them),
Mutualize cmake/configure code as generic reused functions and avoid copy/paste boilerplate,
Rely on CMake as much as possible to make it easy to spot new CMake build options to disable them and other changes.

When using configure it is hard to compare the list of already used options with the new options, one has to read configure's whole output and compare with what's currently used. On the contrary with cmake, one just runs the existing command, then go to the build dir and run ccmake, and see what's enabled and should not, and report the difference to the build script.

illwieckz · 2024-11-18T03:06:40Z

windows-amd64-mingw and windows-i686-mingw engine both build and run.

illwieckz · 2024-11-18T18:10:38Z

For some reasons libpng now provides png.framework on macOS and the engine build looks for it instead of the static library (even with USE_STATIC_LIBS), so I made generic the copy of any *.framework from deps. I don't know if we prefer static linking or dynamic linking on macOS, static linking allows further LTO optimizations so we may prefer that.

illwieckz · 2024-11-18T18:48:27Z

linux-i686-default and macos-amd64-default engine both build and run.

slipher · 2025-04-03T22:49:45Z

Rebased following #1629. Resulting build.sh text is the same as the original modulo NASM_BASEURL.

… use them

slipher · 2025-04-10T20:50:15Z

I tried to test builds before/after to see if the changes broke anything. There was a lot of stuff broken but much of it was also broken without the changes. I opened #1642 with fixes for those issues which are present both before and after. That fixes the GLEW Windows error mentioned in the OP.

I got some issues with the macos bash not liking certain constructs:

The ^^ in the log function. That was a pre-existing bug so I put a fix in More external_deps build fixes #1642.
Evaluating empty arrays. Luckily we don't have any need for variables with spaces, so arrays don't provide any value and can be replaced with regular variables.
The ${@} in the configure_build function seemed to cause an error whenever the function got 0 arguments. This one is really weird as $@ seems like a very standard, ancient shell functionality. I hacked around it for testing with ${@:-}, not sure if this is really write (would it add an empty string arg?) but it seemed to work.

slipher · 2025-04-11T09:04:12Z

I guess the problem is that the old Bash sometimes doesn't distinguished between "undefined" and "empty" variables. Maybe for the configure_build $@ thing we could wrap it in set +u/set -u?

To answer some very old posts from the thread above:

Also I noticed that SDL2 has some audio backends (like pulseaudio), and OpenAL has an SDL2 backend (among others like pulseaudio), so we may check we only build them once. Either we use OpenAL on native audio backend directly and we make sure SDL2 doesn't build any native audio backend, either we use OpenAL on SDL2 and make sure that's the only OpenAL backend to be built and that SDL2 has the required native audio backends being built.

We aren't using SDL audio.

Regarding static vs. dynamic libs, I guess static is better for releases since the linker has the opportunity to remove unused code, reducing the file size. Dynamic may be better for development since then you don't have to link anything so the build could be faster. MSVC libs have to be dynamic because we can't produce static libs in its format. Fortuitously, MSVC is the one platform which is never used for releases, only development. So my conclusion would be static libs are preferable for everything but MSVC.

slipher · 2025-04-11T17:25:53Z

After applying some fixes for the old Bash compatibility and the fix for pkgconfig from #1642, I successfully built and tested the Mac amd64 deps and client with the latest toolchain and OS on my arm64 Mac machine.

slipher · 2025-04-19T05:23:37Z

external_deps/build.sh

-	make
-	make install
+
+	cmake_build \


There's a regression with the PNG build for MSVC. It produces an unwanted file at lib/libpng.dll. This is not a DLL, but a symlink to a mingw-format archive (useless for MSVC). In addition to this file being undesirable, something about the symlink causes tar to return an error code in the package step.

slipher · 2025-04-19T06:13:21Z

external_deps/build.sh

-		CFLAGS="${CFLAGS} -O3" ./configure --prefix="${PREFIX}" --libdir="${PREFIX}/lib" --static --const
-		make
-		make install
+		windows-*-*)


zlib now produces both static and shared libs for mingw platforms. Previously it only produced DLLs. We should make only one or the other.

slipher · 2025-04-20T03:56:38Z

Another bug that is present in both master and this branch: sdl2-config.cmake may contain absolute paths referencing the build directory. On master it affects only Linux platforms. On this branch it only affects MinGW platforms. The presence of that file prevents an MSYS2 Unvanquished build from configuring. But deleting it doesn't work either as they changed around some stuff with SDL2main...

slipher · 2025-04-20T04:45:25Z

There's a big problem with the .cmake configuration files declaring MinGW-style import libs (.dll.a) in MSVC packages. This prevents MSVC builds from configuring. On master the import lib problem affects just jpeg; here it also affects vorbis, curl, opus, and png. Probably 7548f02 was a mistake and should be reverted (my bad for not properly testing). But then we would still have problems with SDL - the old finding code does not handle it properly and we would need the sdl2-config.cmake to get the right set of libs.

slipher · 2025-04-20T08:43:10Z

Added a new commit external_deps: don't munge SDL config to #1642. With that plus my GLEW fix from the same PR on top of your changes, I get MinGW packages that build and run under MSYS2.

slipher · 2025-04-21T10:55:14Z

cmake/cross-toolchain-mingw32.cmake

@@ -8,7 +8,7 @@ set( CMAKE_CXX_COMPILER i686-w64-mingw32-g++ )
 set( CMAKE_RC_COMPILER i686-w64-mingw32-windres )

 # Target prefix
-set( CMAKE_FIND_ROOT_PATH /usr/i686-w64-mingw32 )
+set( CMAKE_FIND_ROOT_PATH /usr/i686-w64-mingw32;${CMAKE_PREFIX_PATH} )


Surely this doesn't do anything, as the toolchain file is executed before CMakeLists.txt, so CMAKE_PREFIX_PATH wouldn't be set (unless you set it on the command line.)

Oh it's for building external_deps, not compiling the game. Could use a comment

slipher · 2025-04-21T11:02:00Z

external_deps/build.sh

+	case "${PLATFORM}" in
+	windows-*-*)
+		# CMake looks for libSDL2.a and aborts if missing if this file exists:
+		rm -rf "${PKG_PREFIX}/lib/cmake/SDL2/SDL2staticTargets.cmake"


What's this? I don't see a file like this anywhere, and if I revert this commit it does not appear.

For me, this file came up and sabotaged the build not for Windows, but for Linux targets by injecting a -lsamplerate flag. Anyway on the slipher/deps-combined branch I reverted that commit and did a proper fix by configuring the SDL build not to use the dependency.

slipher · 2025-04-21T11:09:54Z

external_deps/build.sh

+		-DCURL_USE_MBEDTLS=OFF \
+		-DCURL_USE_NSS=OFF \
+		-DCURL_USE_OPENSSL=OFF \
+		-DCURL_USE_WOLFSSL=ON \


slipher · 2025-04-21T22:47:44Z

The zlib build is not usable for MSVC because it produces a zconf.h including <unistd.h>.

slipher · 2025-04-22T05:33:33Z

For MSVC platforms, the resulting JPEG, WebP, and Curl DLLs have a libgcc DLL dependency. This problem is actually already present with JPEG in the version 10 deps package.
So you currently cannot run a client built with Visual Studio if you do not have MinGW in your PATH. With webp and curl it is a new regression.

slipher · 2025-04-22T22:37:20Z

I have pushed a branch named slipher/deps-combined which fixes all issues known to me. It contains the commits from #1642, #1643, this PR, and several more on top of that.

illwieckz marked this pull request as draft November 11, 2024 16:57

illwieckz force-pushed the illwieckz/deps branch from cc59996 to bd1e318 Compare November 11, 2024 18:24

illwieckz force-pushed the illwieckz/deps branch from 03961c7 to c537fe9 Compare November 13, 2024 03:03

illwieckz force-pushed the illwieckz/deps branch 3 times, most recently from 5860e20 to c73312c Compare November 18, 2024 02:37

illwieckz force-pushed the illwieckz/deps branch from 2773253 to 2dd223e Compare November 18, 2024 18:44

illwieckz mentioned this pull request Mar 24, 2025

ci: use submodules for freetype #1585

Draft

illwieckz force-pushed the illwieckz/deps branch 2 times, most recently from 26792b6 to dbf2de7 Compare March 26, 2025 04:57

slipher mentioned this pull request Mar 28, 2025

Print error when curl command not found #1630

Open

slipher force-pushed the illwieckz/deps branch from 984e303 to de76382 Compare April 3, 2025 22:48

illwieckz added 11 commits April 10, 2025 13:00

cmake: make MinGW toolchains also look for CMAKE_PREFIX_PATH

0b452ba

cmake: copy all macOS frameworks from deps

394e7fd

external_deps: fix SDL2 windows build

d343f0d

external_deps: refactor and unify cmake and configure usage

1f17016

external_deps: mutualize some MinGW and MSVC code

fa7b0ad

cmake,external_deps: install and look for macos frameworks in lib/

d4a1300

external_deps: introduce cmake_build and configure_build wrappers and…

2672541

… use them

external_deps: revert ogg from cmake to configure (it does not install)

56b6135

external_deps: use -O2 and -fPIC for all packages

f439a2f

external_deps: bump lib versions

b9b25d4

cmake,external_deps: bump deps version

caf3648

slipher force-pushed the illwieckz/deps branch from de76382 to caf3648 Compare April 10, 2025 18:15

slipher mentioned this pull request Apr 10, 2025

More external_deps build fixes #1642

Open

slipher reviewed Apr 19, 2025

View reviewed changes

slipher reviewed Apr 21, 2025

View reviewed changes

external_deps/build.sh

-DCURL_USE_MBEDTLS=OFF \

-DCURL_USE_NSS=OFF \

-DCURL_USE_OPENSSL=OFF \

-DCURL_USE_WOLFSSL=ON \

Copy link

Member

slipher Apr 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why on?

external_deps: revamp lib build, add cmake and configure wrapper, update lib feature disablement, update lib versions #1433

Are you sure you want to change the base?

external_deps: revamp lib build, add cmake and configure wrapper, update lib feature disablement, update lib versions #1433

Conversation

illwieckz commented Nov 11, 2024 • edited Loading

illwieckz commented Nov 11, 2024 • edited Loading

illwieckz commented Nov 11, 2024 • edited Loading

illwieckz commented Nov 11, 2024 • edited Loading

illwieckz commented Nov 11, 2024 • edited Loading

illwieckz commented Nov 11, 2024

illwieckz commented Nov 11, 2024

illwieckz commented Nov 11, 2024

illwieckz commented Nov 11, 2024

illwieckz commented Nov 11, 2024

illwieckz commented Nov 11, 2024

illwieckz commented Nov 11, 2024

slipher commented Nov 11, 2024

illwieckz commented Nov 16, 2024

slipher commented Nov 16, 2024

illwieckz commented Nov 16, 2024

DolceTriade commented Nov 16, 2024

illwieckz commented Nov 18, 2024 • edited Loading

illwieckz commented Nov 18, 2024

illwieckz commented Nov 18, 2024

illwieckz commented Nov 18, 2024

slipher commented Apr 3, 2025

slipher commented Apr 10, 2025 • edited Loading

slipher commented Apr 11, 2025

slipher commented Apr 11, 2025

slipher Apr 19, 2025 • edited Loading

Choose a reason for hiding this comment

slipher Apr 19, 2025 • edited Loading

Choose a reason for hiding this comment

slipher commented Apr 20, 2025 • edited Loading

slipher commented Apr 20, 2025

slipher commented Apr 20, 2025

slipher Apr 21, 2025

Choose a reason for hiding this comment

slipher Apr 21, 2025

Choose a reason for hiding this comment

slipher Apr 21, 2025

Choose a reason for hiding this comment

slipher Apr 23, 2025

Choose a reason for hiding this comment

slipher Apr 21, 2025

Choose a reason for hiding this comment

slipher commented Apr 21, 2025

slipher commented Apr 22, 2025

slipher commented Apr 22, 2025

illwieckz commented Nov 11, 2024 •

edited

Loading

illwieckz commented Nov 11, 2024 •

edited

Loading

illwieckz commented Nov 11, 2024 •

edited

Loading

illwieckz commented Nov 11, 2024 •

edited

Loading

illwieckz commented Nov 11, 2024 •

edited

Loading

illwieckz commented Nov 18, 2024 •

edited

Loading

slipher commented Apr 10, 2025 •

edited

Loading

slipher Apr 19, 2025 •

edited

Loading

slipher Apr 19, 2025 •

edited

Loading

slipher commented Apr 20, 2025 •

edited

Loading