Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Intel GPU] Symbol conflict between libm.lib and ucrt.lib for Intel GPU on Windows #134989

Open
EikanWang opened this issue Sep 3, 2024 · 4 comments · May be fixed by shibatch/sleef#603
Open
Assignees
Labels
module: xpu Intel XPU related issues triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@EikanWang
Copy link
Collaborator

EikanWang commented Sep 3, 2024

🐛 Describe the bug

Currently, building Intel GPU for Windows requires sourcing Intel GPU development bundle for Windows to setup development environment for PyTorch. But the bundle contains libm.lib to support C99 and contains math functions. and the And the lib will be added to Lib path, while it conflicts with Windows native ucrt.lib. The error message may be as follows.

ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: ldexp already defined in libm.lib(ldexp_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: copysign already defined in libm.lib(copysign_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: copysignf already defined in libm.lib(copysignf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: expf already defined in libm.lib(expf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: tanhf already defined in libm.lib(tanhf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: sinf already defined in libm.lib(sinf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: logf already defined in libm.lib(logf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: sqrtf already defined in libm.lib(sqrtf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: cosf already defined in libm.lib(cosf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: floorf already defined in libm.lib(floorf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: ceil already defined in libm.lib(ceil_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: ceilf already defined in libm.lib(ceilf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: floor already defined in libm.lib(floor_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: log2f already defined in libm.lib(log2f_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: truncf already defined in libm.lib(truncf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: acosf already defined in libm.lib(acosf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: asinf already defined in libm.lib(asinf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: atan2f already defined in libm.lib(atan2f_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: atanf already defined in libm.lib(atanf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: coshf already defined in libm.lib(coshf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: log10f already defined in libm.lib(log10f_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: powf already defined in libm.lib(powf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: sinhf already defined in libm.lib(sinhf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: tanf already defined in libm.lib(tanf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: fmodf already defined in libm.lib(fmodf_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: round already defined in libm.lib(round_iface_c99.obj)
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: roundf already defined in libm.lib(roundf_iface_c99.obj)
   Creating library lib\torch_cpu.lib and object lib\torch_cpu.exp
bin\torch_cpu.dll : fatal error LNK1169: one or more multiply defined symbols found
ninja: build stopped: subcommand failed.

Versions

Collecting environment information...
PyTorch version: 2.5.0a0+git3f3774a
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Enterprise
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.24.1
Libc version: N/A

Python version: 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:29:51) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22631-SP0
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=2400
DeviceID=CPU0
Family=207
L2CacheSize=14336
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=2400
Name=12th Gen Intel(R) Core(TM) i9-12900
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] numpy==2.1.0
[pip3] optree==0.12.1
[pip3] torch==2.5.0a0+git3f3774a
[conda] mkl-include 2024.2.1 pypi_0 pypi
[conda] mkl-static 2024.2.1 pypi_0 pypi
[conda] numpy 2.1.0 pypi_0 pypi
[conda] optree 0.12.1 pypi_0 pypi
[conda] torch 2.5.0a0+git3f3774a pypi_0 pypi

cc @gujinghui @fengyuan14 @guangyey

@EikanWang EikanWang added module: xpu Intel XPU related issues triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Sep 3, 2024
@EikanWang EikanWang self-assigned this Sep 3, 2024
@EikanWang
Copy link
Collaborator Author

Regarding this issue, we can workaround it to pass the linking process by setting some cmake environment variables as follows.

set CMAKE_SHARED_LINKER_FLAGS=/FORCE:MULTIPLE
set CMAKE_MODULE_LINKER_FLAGS=/FORCE:MULTIPLE
set CMAKE_EXE_LINKER_FLAGS=/FORCE:MULTIPLE

And eventually, we will update the intel GPU bundle to fix it.

@xuhancn xuhancn self-assigned this Oct 31, 2024
@xuhancn
Copy link
Collaborator

xuhancn commented Oct 31, 2024

Let me do some analysis here.

Firstly, this issue is that, compiler link stage, it found the same math function have two implenmet. Let's focus on ldexp function, and do some debug.

ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: ldexp already defined in libm.lib(ldexp_iface_c99.obj)

From the error message, we found:

  1. ldexp in api-ms-win-crt-math-l1-1-0.dll.
  2. ldexp in libm.lib, and which is assembly from ldexp_iface_c99.obj.

Let's validate 1, we can use dependency walker to check it:
image
Yes, it is implenment by Microsoft.

Let's validate 2, and here is another question, which libm.lib involved into this issue?
a. sleef library's.
b. Intel compiler(icx)'s
We mentioned that, the second ldexp is assembly from ldexp_iface_c99.obj.
For sleef, we can find that sleef code didn't has a source code named ldexp_iface_c99, repo: https://github.com/shibatch/sleef.
For icx's, we can also validate it.
image
We can find intel compiler has a libm.lib. Let's use dumpbin tool to decode this library file. We can scaned ldexp_iface_c99.obj from it.

So far, we can make a conclusion, icx's libm have same functions implenmet as MSFT's lib.


Additional analysis for Intel compiler. Acturally, Intel compiler for performance purpose, it would do some magic operation, to replace system's libm to its.

When I enable torch inductor on Windows CPU, I meet this behavior. Below image show the dependency.
Image
We can comfirm additional icx related libs libmmd.dll and svml_dispmd.dll are involeved. To compatible to this behavior, I add some pre_load code:

"""
Intel Compiler implenmented more math libraries than clang, for performance proposal.
We need preload them like openmp library.
"""
preload_list = [
"libiomp5md.dll", # openmp
"svml_dispmd.dll", # svml library
"libmmd.dll", # libm
]

I have experience to handle Intel compiler's behaior.


In currently PyTorch XPU build case, Intel compiler only work as host compiler and dedicated for SYCL code compiling. The major code is compiled by MSVC.
Some PyTorch code was compiled by MSVC and linked to MSFT implenmented ldexp. SYCL code was compiled by Intel Compiler and linked to Intel implenmented ldexp. So, we linked to mutiple ldexp.
Intel compiler work as host compiler, need to compatible to MSVC. We need some build option to tell intel compiler don't do the magic optimization operations, and use system(MSFT) implenmented libm functions.

@xuhancn
Copy link
Collaborator

xuhancn commented Nov 2, 2024

Continued update:

  1. We find the Intel compiler build option to disable its libm: https://www.intel.com/content/www/us/en/docs/dpcpp-cpp-compiler/developer-guide-reference/2024-2/no-intel-lib-qno-intel-lib.html . But it is doesn't work.
    Because of the torch_cpu.dll is built by MSVC, but not by intel compiler.

  2. For the libm bringed in, I did some research.
    a. Some PyTorch third party submodules, such as sleef, xnnpack. Their cmake files may call find_library(m) to search libm. Usually, they can't find the libm on Windows MSVC environment.
    b. When we build XPU version of PyTorch, we need to source Intel compiler environment. And it will bring its libm to build environment. So, step a on XPU build environment would bring in libm into dependency.

Acturally we can't forbid all current/further third party libraries call find_library(m) on Windows.
Setup /FORCE:MULTIPLE should be the only way to solve the issue. I will submit a PR to automatic configure it for PyTorch XPU build.

Original command:

set CMAKE_SHARED_LINKER_FLAGS=/FORCE:MULTIPLE
set CMAKE_MODULE_LINKER_FLAGS=/FORCE:MULTIPLE
set CMAKE_EXE_LINKER_FLAGS=/FORCE:MULTIPLE

@xuhancn
Copy link
Collaborator

xuhancn commented Nov 13, 2024

Continued update:

  1. We find the Intel compiler build option to disable its libm: https://www.intel.com/content/www/us/en/docs/dpcpp-cpp-compiler/developer-guide-reference/2024-2/no-intel-lib-qno-intel-lib.html . But it is doesn't work.
    Because of the torch_cpu.dll is built by MSVC, but not by intel compiler.
  2. For the libm bringed in, I did some research.
    a. Some PyTorch third party submodules, such as sleef, xnnpack. Their cmake files may call find_library(m) to search libm. Usually, they can't find the libm on Windows MSVC environment.
    b. When we build XPU version of PyTorch, we need to source Intel compiler environment. And it will bring its libm to build environment. So, step a on XPU build environment would bring in libm into dependency.

Acturally we can't forbid all current/further third party libraries call find_library(m) on Windows. Setup /FORCE:MULTIPLE should be the only way to solve the issue. I will submit a PR to automatic configure it for PyTorch XPU build.

Original command:

set CMAKE_SHARED_LINKER_FLAGS=/FORCE:MULTIPLE
set CMAKE_MODULE_LINKER_FLAGS=/FORCE:MULTIPLE
set CMAKE_EXE_LINKER_FLAGS=/FORCE:MULTIPLE

Discussed with @EikanWang , add these link flag will hide potential same function name with different parameter issue. So will will withdraw this solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module: xpu Intel XPU related issues triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
Status: No status
Development

Successfully merging a pull request may close this issue.

2 participants