MinGW ld.lld: error: too many exported symbols

[kmilos]

This is from building 3.1.7 in CLANG64 environment (Clang 16.0.1 currently):

FAILED: src/python/PyImath/libPyImath_Python3_10-3_1.dll src/python/PyImath/libPyImath_Python3_10-3_
1.dll.a
cmd.exe /C "cd . && C:\msys64\clang64\bin\clang++.exe -march=nocona -msahf -mtune=generic -O2 -pipe
-O3 -DNDEBUG  -pipe -shared -o src\python\PyImath\libPyImath_Python3_10-3_1.dll -Wl,--out-implib,src
\python\PyImath\libPyImath_Python3_10-3_1.dll.a -Wl,--major-image-version,30,--minor-image-version,0
 src/python/PyImath/CMakeFiles/PyImath_Python3_10.dir/PyImath.cpp.obj src/python/PyImath/CMakeFiles/
PyImath_Python3_10.dir/PyImathAutovectorize.cpp.obj src/python/PyImath/CMakeFiles/PyImath_Python3_10
.dir/PyImathBox2Array.cpp.obj src/python/PyImath/CMakeFiles/PyImath_Python3_10.dir/PyImathBox3Array.
cpp.obj src/python/PyImath/CMakeFiles/PyImath_Python3_10.dir/PyImathBox.cpp.obj src/python/PyImath/C
MakeFiles/PyImath_Python3_10.dir/PyImathBufferProtocol.cpp.obj src/python/PyImath/CMakeFiles/PyImath
_Python3_10.dir/PyImathColor3.cpp.obj src/python/PyImath/CMakeFiles/PyImath_Python3_10.dir/PyImathCo
lor4.cpp.obj src/python/PyImath/CMakeFiles/PyImath_Python3_10.dir/PyImathEuler.cpp.obj src/python/Py
Imath/CMakeFiles/PyImath_Python3_10.dir/PyImathFixedArray.cpp.obj src/python/PyImath/CMakeFiles/PyIm
ath_Python3_10.dir/PyImathFrustum.cpp.obj src/python/PyImath/CMakeFiles/PyImath_Python3_10.dir/PyIma
thLine.cpp.obj src/python/PyImath/CMakeFiles/PyImath_Python3_10.dir/PyImathMatrix22.cpp.obj src/pyth
on/PyImath/CMakeFiles/PyImath_Python3_10.dir/PyImathMatrix33.cpp.obj src/python/PyImath/CMakeFiles/P
yImath_Python3_10.dir/PyImathMatrix44.cpp.obj src/python/PyImath/CMakeFiles/PyImath_Python3_10.dir/P
yImathPlane.cpp.obj src/python/PyImath/CMakeFiles/PyImath_Python3_10.dir/PyImathQuat.cpp.obj src/pyt
hon/PyImath/CMakeFiles/PyImath_Python3_10.dir/PyImathRandom.cpp.obj src/python/PyImath/CMakeFiles/Py
Imath_Python3_10.dir/PyImathShear.cpp.obj src/python/PyImath/CMakeFiles/PyImath_Python3_10.dir/PyIma
thStringArray.cpp.obj src/python/PyImath/CMakeFiles/PyImath_Python3_10.dir/PyImathStringTable.cpp.ob
j src/python/PyImath/CMakeFiles/PyImath_Python3_10.dir/PyImathTask.cpp.obj src/python/PyImath/CMakeF
iles/PyImath_Python3_10.dir/PyImathUtil.cpp.obj src/python/PyImath/CMakeFiles/PyImath_Python3_10.dir
/PyImathFixedVArray.cpp.obj src/python/PyImath/CMakeFiles/PyImath_Python3_10.dir/PyImathVec2fd.cpp.o
bj src/python/PyImath/CMakeFiles/PyImath_Python3_10.dir/PyImathVec2si.cpp.obj src/python/PyImath/CMa
keFiles/PyImath_Python3_10.dir/PyImathVec3fd.cpp.obj src/python/PyImath/CMakeFiles/PyImath_Python3_1
0.dir/PyImathVec3siArray.cpp.obj src/python/PyImath/CMakeFiles/PyImath_Python3_10.dir/PyImathVec3si.
cpp.obj src/python/PyImath/CMakeFiles/PyImath_Python3_10.dir/PyImathVec4fd.cpp.obj src/python/PyImat
h/CMakeFiles/PyImath_Python3_10.dir/PyImathVec4siArray.cpp.obj src/python/PyImath/CMakeFiles/PyImath
_Python3_10.dir/PyImathVec4si.cpp.obj  src/Imath/libImath.dll.a  C:/msys64/clang64/lib/libpython3.10
.dll.a  C:/msys64/clang64/lib/libboost_python310-mt.dll.a  -lm  -lkernel32 -luser32 -lgdi32 -lwinspo
ol -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."
ld.lld: error: too many exported symbols (got 70446, max 65535)
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

[meshula]

Is there any way to attach to this thread, a list of the generated symbols? The only way to diagnose this is to discover where the excess symbols come from. It's most likely due to unintentional symbol visibility, such as in-between templates in pybind11 being exported even though they are not intended as API.

It must be specific to the build scenario, because the 64k symbol limit is a windows thing, so there must be a difference between the exported symbols resulting from msvc and clang under mingw.

I'm also not in a set up to check this, unfortunately.

[kmilos]

Is there any way to attach to this thread, a list of the generated symbols?

I'll try to look into it when I find some more time.

there must be a difference between the exported symbols resulting from msvc and clang under mingw

It looks like it's only clang, the gcc build is also fine.

[kmilos]

Poking around, this looks suspect:

Imath/src/python/PyImath/PyImathExport.h

Line 15 in 7355971

#elif defined(_MSC_VER)

Changing that, however to _WIN32 to cover MinGW as well produces a bunch of new errors - a few of them are to do w/ exporting deleted methods (similar to already patched for Clang build w/o Python bindings), but there are bunch of new ones in both Clang and GCC...

[kmilos]

OTOH, defining -DPLATFORM_VISIBILITY_AVAILABLE (as supported by both clang and gcc on MinGW) leads to the same "too many exported symbols" w/ clang...

[kmilos]

-DPLATFORM_VISIBILITY_AVAILABLE -fvisibility=hidden finally works for MinGW Clang.

Looking at the CMake related options of the whole python subfolder, the symbol visibility seems to be handled/set up quite differently to the main library?

[meshula]

the python bindings don't need exhaustive symbol exporting, only enough that the modules can be dloaded by python itself.

that patch you found (https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-imath/0002-fix-openexr-clang-build.patch) probably needs to be applied here.

And if there's more you discovered, then those as well. Can you list the others that come up? (perhaps as an attached text file if it's too huge to post in a message?)

[kmilos]

That's the only one I know, apparently OpenEXR doesn't build w/ Clang without it.

[ZTKlein]

-DPLATFORM_VISIBILITY_AVAILABLE -fvisibility=hidden finally works for MinGW Clang.

Makes sense. These flags in combination with the visibility hidden attribute should lead to the same behavior as using dllexport in a file when compiling on Windows. By default on Windows all symbols are exported, the exception being when dllexport is found in a file, which means that only that which is declared with dllexport is exported. By setting the platform_visibility_available definition, it uses visibility("default") instead, which does not trigger the same behavior; you need the visibility=hidden compile flag to restrict exports to those where it's declared.

One possible fix I'm working on based off of #324 is to see if it causes issues to simply remove the declarations from locations where it's problematic, and assume compilation with the __declspec(dllexport) declarations, rather than rely on flags being set during compile time. I'm currently in the process of getting tests running and setting up a regular linux environment to test compilation there as well using GCC.