A clang64-cross environment

[past-due]

A clang64-cross environment that expects an x86_64 host, but has all the necessary base toolchain + libs to cross-compile for all targets (i686, x86_64, armv7, aarch64).

i.e. Similar to https://github.com/mstorsjo/llvm-mingw, but with the benefit of all of msys2's patches and tweaks and surrounding infrastructure (ex. msys2/setup-msys2 GitHub Action).

Useful for local or hosted CI that doesn't offer appropriate target architecture runners. (i.e. clangarm64 expects to be run on ARM64, but - at least as of 2020 - there were no plans to bring hosted ARM64 Windows environments to GitHub Actions)

---

Theoretically, various other libraries could come from clang32, clang64, clangarm64, but the open-source projects that I contribute to build all dependencies with vcpkg anyway (and this works well with mingw), so a simple clang/llvm cross-compilation msys2 environment would be sufficient as a fantastic start.

Thoughts? Ideas?

[jeremyd2019]

I was thinking about this, and if there are options to clang/lld/etc to override their search paths (and I assume there are), including for compiler-rt, it seems like should be possible to make a mingw-w64-cross-clang type package that consists entirely of scripts that pass the proper args to the host arch's mingw-w64-clang to use the target arch's mingw-w64-compiler-rt, libc++, crt, etc... Such a package could depend on host mingw-w64-clang packages, and target mingw-w64-* packages but that would kind of break the self-contained-ness of each prefix I guess

[jeremyd2019]

clang -resource-dir ?

[mati865]

Oh, had no idea this exists.

[jeremyd2019]

with clang64...

$ clang++ -target aarch64-w64-windows-gnu --sysroot /clangarm64 -resource-dir /clangarm64/lib/clang/12.0.0 -o test.exe test.cpp

works! no idea if anything more complicated would work though.

[past-due]

One concern with depending on the other environments for core libraries (like compiler-rt) is that there's no guarantee the version will match. (An example is recently, where /clang64 had Clang 12 but /clangarm64 still had Clang 11.) That might be asking for trouble - example: https://bugzilla.redhat.com/show_bug.cgi?id=1800705. (Non-core compiler toolchain libs are another matter, though, and I don't see why most couldn't come from the /clang<arch> folder.)

Might it be possible to bootstrap compiling llvm / clang / compiler-rt for all target archs using the /clang64 environment, and then distribute that as the core compiler package for /clang64-cross, ensuring that the versions always match?

[jeremyd2019]

One concern with depending on the other environments for core libraries (like compiler-rt) is that there's no guarantee the version will match. (An example is recently, where /clang64 had Clang 12 but /clangarm64 still had Clang 11.) That might be asking for trouble - example: https://bugzilla.redhat.com/show_bug.cgi?id=1800705. (Non-core compiler toolchain libs are another matter, though, and I don't see why most couldn't come from the /clang folder.)

yes, that is a concern. Other than the fact that clangarm64 is still very experimental and manually built and uploaded, this is not supposed to happen. If it were packaged, the package could be constructed to depend on exact versions between the two environments.

Might it be possible to bootstrap compiling llvm / clang / compiler-rt for all target archs using the /clang64 environment, and then distribute that as the core compiler package for /clang64-cross, ensuring that the versions always match?

yes, but I don't think you'd need to. I think you'd just need a cross-compiler-rt package will all the compiler-rts. It looked like libc++/libunwind were picked up from the sysroot

[jeremyd2019]

(based on https://github.com/msys2/MSYS2-packages/blob/6ba6328319552ee213281309aafab7d0f5a892c2/mingw-w64-cross-clang/PKGBUILD#L145-L150)

#!/bin/bash
_target=aarch64-w64-mingw32
_targetprefix=/clangarm64
_prefix=cross-aarch64/bin
_resourcever="$(basename "$(clang -print-resource-dir)")"

for tool in as cc clang gas gcc; do
  printf "#!/bin/bash\n\n${MINGW_PREFIX}/bin/clang -target ${_target} --sysroot=${_targetprefix} -resource-dir=${_targetprefix}/lib/clang/${_resourcever} \"\$@\"" | tee "${_prefix}/${tool}" "${_prefix}/${_target}-${tool}" > /dev/null
done
for tool in c++ clang++ g++; do
  printf "#!/bin/bash\n\n${MINGW_PREFIX}/bin/clang++ -target ${_target} --sysroot=${_targetprefix} -resource-dir=${_targetprefix}/lib/clang/${_resourcever} \"\$@\"" | tee "${_prefix}/${tool}" "${_prefix}/${_target}-${tool}" > /dev/null
done
for tool in windres; do
  printf "#!/bin/bash\n\nexec -a ${MINGW_PREFIX}/bin/${_target}-${tool} ${MINGW_PREFIX}/bin/${tool} \"\$@\"" | tee "${_prefix}/${tool}" "${_prefix}/${_target}-${tool}" > /dev/null
done

I was able to cross build ntldd with this at least.

I am having issues with windres though, I need to set argv[0] for the wrapper to pick up the proper target, but exec -a doesn't seem to be working. Also, even if I did it would then try to call aarch64-w64-mingw32-clang in the same directory as itself, which doesn't exist in /clang64/bin/ and gets an error in the cross-aarch64/bin directory as a native program is trying to launch a bash script.

[jeremyd2019]

Well it's possible to set argv[0] in Windows, cygwin just doesn't...

#include <windows.h>
#include <stdio.h>

#ifndef REAL_COMMAND
# define REAL_COMMAND L"llvm-windres"
#endif

int wmain(int argc, WCHAR ** argv)
{
	STARTUPINFO si = {0};
	PROCESS_INFORMATION pi = {0};
	WCHAR wszToLaunch[MAX_PATH] = {0};
	DWORD dwRet = 0;

	si.cb = sizeof(si);
	dwRet = SearchPathW(NULL, REAL_COMMAND, L".exe", ARRAYSIZE(wszToLaunch), wszToLaunch, NULL);
	if (dwRet == 0 || dwRet > ARRAYSIZE(wszToLaunch))
	{
		fprintf(stderr, "SearchPath failed %lu\n", GetLastError());
		return 1;
	}

	if (!CreateProcessW(wszToLaunch, GetCommandLineW(), NULL, NULL, TRUE, 0, NULL, NULL, &si, &pi))
	{
		fprintf(stderr, "CreateProcess failed %lu\n", GetLastError());
		return 2;
	}

	CloseHandle(pi.hThread);
	WaitForSingleObject(pi.hProcess, INFINITE);
	if (!GetExitCodeProcess(pi.hProcess, &dwRet))
	{
		fprintf(stderr, "GetExitCodeProcess failed %lu\n", GetLastError());
		CloseHandle(pi.hProcess);
		return 3;
	}
	CloseHandle(pi.hProcess);
	return (int)dwRet;
}

But it still fails to find /clang64/bin/aarch64-w64-mingw32-clang

[mstorsjo]

Yes, in principle one can pass arbitrary values as argv[0] on windows too - but it's a bit tricky at times. E.g. clang's init routines try to figure out exact where the clang executable resides, and that also overwrites argv[0] with the canonical executable name. (That's not relevant here though, but I've considered similar approaches for some aspects of my wrappers at some point.)

[mstorsjo]

Also regarding windres; before spending too much time on the current setup, I'd recommend backporting the commits for adding llvm-windres if you didn't already, as that's the path forward for the future anyway.

With llvm-windres, you can pass a generic triple to the -F/--target option, so you don't need to mess around with the argv[0] prefix approach. For compatibility with GNU windres, the -F option only would take arguments like pe-x86-64 that libbfd knows about, but in llvm-windres I've added a fallback to interpreting the argument as a regular triple if it doesn't match the known BFD format names. (The windres wrapper tool in llvm-mingw doesn't support that fallback, but it should be trivial to add it if you think it's a good idea.)

[jeremyd2019]

Also regarding windres; before spending too much time on the current setup, I'd recommend backporting the commits for adding llvm-windres if you didn't already, as that's the path forward for the future anyway.

Yes we started looking at that in msys2/CLANG-packages#44, but it failed on python's rc where the old wrapper worked. I haven't had a chance to try to extract a small reproducing case yet.

[mstorsjo]

Oh, thanks for letting me know about that. I've got a setup where I build python, I can try to build that with llvm-windres in place and see if I can figure out what's wrong.

[jeremyd2019]

I threw together a PKGBUILD based on my ideas: #8762

[msink]

Did I get it correctly that it will be a new package ${MINGW_PACKAGE_PREFIX}-clang-cross mutually exclusive with ${MINGW_PACKAGE_PREFIX}-clang using same mechanism as in #8469?
And excepting windres it really will be just bash scripts calling host tools from ../clang64/bin.

[msink]

If it will be target package most likely packaging tools will not notice anything and just work.
Your comment in msys2/MSYS2-packages#2440 (comment)

[mati865]

The cannot be bash wrappers since native build systems won't be able to call them. They have to be executables.

And they cannot conflict because some packages will need libraries from clang or llvm.

[msink]

Same as in #8464 they can be marked as both conflicts and provides:

provides=("${MINGW_PACKAGE_PREFIX}-${_realname}")
conflicts=("${MINGW_PACKAGE_PREFIX}-${_realname}")

About native wrappers - AFAIK it's not so hard to compile any script to executable,
but if it will be target package - it will be a bit weird - host executables in target directory...

[mati865]

I had typo in my comment. We need libraries provided by mingw-w64-clang-aarch64-clang and mingw-w64-clang-aarch64-llvm among the others so they cannot conflict.

[jeremyd2019]

The cannot be bash wrappers since native build systems won't be able to call them. They have to be executables.

I was afraid of that, once I saw that windres wrapper couldn't call the bash clang wrapper.

[jeremyd2019]

If we need native wrappers, we're going to wind up with something pretty similar to llvm-mingw. Is there really any benefit to reinventing that here? If so, maybe starting with @mstorsjo's wrappers would be a good idea

[mstorsjo]

Well there's not much inherent extra value in my wrappers compared to what you're adding here; my wrappers set defaults for -fuse-ld, -rtlib and -stdlib, because when cross compiling from a unix, I don't want the clang executable to be built with defaults that only works for the cross target.

[mstorsjo]

Actually on second thought about this (reinventing new wrappers or reusing ones from llvm-mingw), I realized you might want to consider wrappers that are native executables instead of shell scripts after all - when building things with make, where the tool launching the compiler is an msys tool (make, bash etc) then shell scripts work fine, but if building something with e.g. ninja, cmake or meson/python, then pure win32 executables try to call the compiler, and then only native wrappers would work.

So in that case, my wrappers might be a decent starting point, but adjusted with different logic (setting less defaults as they're hardcoded, different logic for figuring out the sysroot).

[jeremyd2019]

I was thinking about this some more, and if there are native wrappers in the 'host' prefix, they will need some way to get the path to the 'target' prefix. I don't know how that would best be accomplished... calling out to cygpath and capturing the output (and assuming we're running from an msys2 shell where cygpath would be on the PATH)? assuming it's under basenamedirname(argv[0])/../../?

[mstorsjo]

Out of those, I'd recommend/prefer something that just looks for e.g. dirname(argv[0])/../../clangarm64/aarch64-w64-mingw32 (if I understand correctly what you mean); you'd want to execute as few extra commands in the wrapper as possible in order to not make it unnecessarily slow.

[jeremyd2019]

yeah, I meant dirname... pseudocode anyway

[jeremyd2019]

I finally came back to this, still in #8762. I went ahead and built/packaged compiler-rt for the 'foreign' windows arches rather than messing around with -resource-dir. This may or may not be sufficient, because I only built the 'builtins' lib - the profiler lib is also present on native clangarm64 but requires the builtins lib to already be present, apparently.

I had to 'resurrect' a dlltool and windres wrapper, because the generic llvm-wrapper doesn't work with them, as they overwrite their argv[0]. Also, there was a bit of annoyance with having to pass a sysroot to windres: apparently something is eating backslashes from the --preprocessor-arg, I'm guessing for compatibility with GNU windres (which is known to suck in regards to its preprocessor options). I went with replacing backslashes with forward slashes, to avoid having to allocate more memory.

I left the defaults in clang-target-wrapper for things like lld/libunwind/rtlib/libc++. I think they may be necessary if these are to work with the MINGW*/UCRT* tools as well (there's no guarantee that they have to though, and it may be simpler to declare that you must use CLANG* with this, and remove those defaults).

[mstorsjo]

Also, there was a bit of annoyance with having to pass a sysroot to windres: apparently something is eating backslashes from the --preprocessor-arg, I'm guessing for compatibility with GNU windres (which is known to suck in regards to its preprocessor options). I went with replacing backslashes with forward slashes, to avoid having to allocate more memory.

If clang can find the sysroot automatically based on the triple (e.g. as <clang-binary>/../${arch}-w64-mingw32) it shouldn't be necessary to specify it manually at all.

But indeed, the --preprocessor-arg options are tricky with respect to backslash handling - GNU windres doesn't re-quote things with respect to backslashes and double quotes when passing arguments to the subshell, so they have to be double quoted - and llvm-windres follows. See https://sourceware.org/bugzilla/show_bug.cgi?id=27843 for a report on that matter.

[jeremyd2019]

Unfortunately no, the sysroot parameter works out as <clang-binary>/../../${MINGW_PREFIX} (for the foreign $MINGW_PREFIX). Clang then seems fine to also find the mingw-w64 headers and crt under the ${arch}-w64-mingw32 subdirectory of that.

Eg.

$ aarch64-w64-mingw32-windres -### -O coff /dev/null /dev/null --preprocessor-arg -v
 "clang" "--driver-mode=gcc" "-target" "aarch64-w64-mingw32" "-E" "-xc" "-DRC_INVOKED" "nul" "-o" "C:/msys64/tmp/preproc-1d6515.rc" "--sysroot=C:/msys64/clang64/bin/../../clangarm64" "-v"

$ aarch64-w64-mingw32-windres -O coff /dev/null /dev/null --preprocessor-arg -v
...
#include "..." search starts here:
#include <...> search starts here:
 C:/msys64/clang64/lib/clang/13.0.0/include
 C:/msys64/clang64/bin/../../clangarm64/aarch64-w64-mingw32/include
 C:/msys64/clang64/bin/../../clangarm64/include
End of search list.

[mstorsjo]

Ah, I see - yes, in a layout like that, I can see that you need to specify the sysroot manually.

[jeremyd2019]

Resurrecting this old discussion, it sounds like something like this could be useful for Git for Windows. /cc @dscho @dennisameling

My concern with #8762 is the cross-repo dependencies. You need the target mingw-w64-headers-git to build the target compiler-rt, and you do need at least a few core libraries for the cross-compilers to be useful. I'm pretty sure no cross-repo dependencies like that currently exist (with the limited exception of dependencies on msys packages), and that autobuild does not handle such a case.

[dscho]

Thank you for your consideration!

In Git for Windows' case, however, I think we really need to lean hard in the GCC direction because we actually do need not only a MINGW side but also an MSYS side of things, and as Corinna pointed out, Cygwin will first need a working MINGW variant of GCC targeting Windows/ARM64, and then hopes to build on that to provide the Cygwin variant (which would give rise to the MSYS one), too.

[mati865]

I think it will take years to have good support for AArch64 Winows in GCC with assumption that somebody is already working on it (haven't heard of such effort yet).

I think it'd be easier to make Cygwin build with Clang.

[dscho]

@mati865 I have not heard of anybody compiling Cygwin programs using Clang. It might be an even bigger effort than AArch64 Windows support in GCC.

I am not saying you're wrong. I'm saying that it is currently totally unclear to me what is the best way to get Cygwin to build natively on Windows/ARM64. I'm still unconvinced that the Clang route has a chance to succeed.

[mstorsjo]

I guess the most relevant intermediate step would be if someone would try out using Clang for current Cygwin on x86, and fix any kinks (or regressions) in that setup. Clang and LLVM does have code for handling the Cygwin target, but I don't think it has been maintained much for the last couple of years (if it ever was well working before that) though. Then secondly, once the compiler for targeting it works, one would need to be able to re-bootstrap the whole cygwin runtime from scratch with that. (That would involve also building either libstdc++, which is kinda tied to gcc, or switching to libc++ on cygwin too.)

From the point of view of someone quite familiar with the Clang/LLVM internals relating to mingw, I think getting the Cygwin target in the compiler up to working state today would mostly consist of trying things out and possibly fixing conditionals relating to picking the right (unix/mingw/msvc) behaviour for different details. (In many cases, there are calls like isWindowsGNUEnvironment() (which means mingw) which may need to be changed into isOSCygMing() or vice versa.)

[jeremyd2019]

I recently came across this blog post: https://blogs.gentoo.org/mgorny/2022/10/07/clang-in-gentoo-now-sets-default-runtimes-via-config-file/

Perhaps these config files can be used instead of wrappers passing different "default" arguments for cross-compiling? And/or instead of patches that change defaults in mingw-w64-clang package? It sounds like there is new behavior regarding these config files coming in clang 16.x (which gentoo partially backported to 15.x) that might make that a good opportunity to explore the possibilities there.

[Biswa96]

Just to make sure, does that mean instead of passing various target options in wrapper file the same options will be listed in single config file? It sounds like toolchain file used in cmake or meson, right?

[jeremyd2019]

That's what I was thinking reading that blog post, but I haven't investigated whether that would actually work or not. I just posted this here to put the idea on other people's radar, and point out that clang 16 changed the config file behavior in a way that seems like it could be useful for cross compiler configuration.