Error in installing

[ziyang-arch]

There are two errors occurred when I tried to install this lib:

1. the cmake.__file__ in line 81 of setup.py is None so it cause a TypeError when it's used as the cmake path. I can let it bypass this part and use #search in path to find the cmake in my system.
2. After fixing the cmake path, it shows fatal error: filesystem: No such file or directory.

My system spec is:
Linux 3.10.0-1160.102.1.x86_64 x86_64 x86_64 GNU/Linux
CUDA 12.3
GPU: Tesla T4
cmake 3.28.0
GCC 11.2.1 devtoolset-11

[timmoon10]

I think this error is happening because you do not have the cmake module installed but do have a directory called cmake in your path. Calling import cmake loads the wrong thing and cmake.__file__ is None:

TransformerEngine/setup.py

Line 77 in 868c7d3

import cmake

#888 should fix this error.

If you are still running into errors after this, it could be that the build system can't find the CMake executable. In this case you should add the executable to you PATH or install CMake with pip install cmake.

[ziyang-arch]

Thank you!
The first error can be fixed in various ways.
My major difficulty is the second error. According to #459, GCC after 8.1 should have filesystem header file.
In my case, the compiler cannot find this header file with GCC 11.2.1

[1/9] Building CXX object common/CMakeFiles/transformer_engine.dir/util/cuda_runtime.cpp.o FAILED: common/CMakeFiles/transformer_engine.dir/util/cuda_runtime.cpp.o /usr/bin/c++ -Dtransformer_engine_EXPORTS -I/**/TransformerEngine/transformer_engine -I/**/TransformerEngine/transformer_engine/common/include -I/**/TransformerEngine/transformer_engine/common/../../3rdparty/cudnn-frontend/include -I/**TransformerEngine/build/cmake/common/string_headers -isystem /usr/local/cuda-12.3/targets/x86_64-linux/include -O3 -DNDEBUG -std=gnu++1y -fPIC -MD -MT common/CMakeFiles/transformer_engine.dir/util/cuda_runtime.cpp.o -MF common/CMakeFiles/transformer_engine.dir/util/cuda_runtime.cpp.o.d -o common/CMakeFiles/transformer_engine.dir/util/cuda_runtime.cpp.o -c /**/TransformerEngine/transformer_engine/common/util/cuda_runtime.cpp /**/TransformerEngine/transformer_engine/common/util/cuda_runtime.cpp:7:22: fatal error: filesystem: No such file or directory #include <filesystem> ^ compilation terminated. [2/9] Building CXX object common/CMakeFiles/transformer_engine.dir/util/cuda_driver.cpp.o FAILED: common/CMakeFiles/transformer_engine.dir/util/cuda_driver.cpp.o /usr/bin/c++ -Dtransformer_engine_EXPORTS -I/**/TransformerEngine/transformer_engine -I/**/TransformerEngine/transformer_engine/common/include -I/**/TransformerEngine/transformer_engine/common/../../3rdparty/cudnn-frontend/include -I/**/TransformerEngine/build/cmake/common/string_headers -isystem /usr/local/cuda-12.3/targets/x86_64-linux/include -O3 -DNDEBUG -std=gnu++1y -fPIC -MD -MT common/CMakeFiles/transformer_engine.dir/util/cuda_driver.cpp.o -MF common/CMakeFiles/transformer_engine.dir/util/cuda_driver.cpp.o.d -o common/CMakeFiles/transformer_engine.dir/util/cuda_driver.cpp.o -c /**/TransformerEngine/transformer_engine/common/util/cuda_driver.cpp /**/TransformerEngine/transformer_engine/common/util/cuda_driver.cpp:8:22: fatal error: filesystem: No such file or directory #include <filesystem> ^ compilation terminated. [3/9] Building CXX object common/CMakeFiles/transformer_engine.dir/util/system.cpp.o FAILED: common/CMakeFiles/transformer_engine.dir/util/system.cpp.o /usr/bin/c++ -Dtransformer_engine_EXPORTS -I/**/TransformerEngine/transformer_engine -I/**/TransformerEngine/transformer_engine/common/include -I/**/TransformerEngine/transformer_engine/common/../../3rdparty/cudnn-frontend/include -I/**/TransformerEngine/build/cmake/common/string_headers -isystem /usr/local/cuda-12.3/targets/x86_64-linux/include -O3 -DNDEBUG -std=gnu++1y -fPIC -MD -MT common/CMakeFiles/transformer_engine.dir/util/system.cpp.o -MF common/CMakeFiles/transformer_engine.dir/util/system.cpp.o.d -o common/CMakeFiles/transformer_engine.dir/util/system.cpp.o -c /**/TransformerEngine/transformer_engine/common/util/system.cpp /**/TransformerEngine/transformer_engine/common/util/system.cpp:9:22: fatal error: filesystem: No such file or directory #include <filesystem> ^ compilation terminated. [4/9] Building CXX object common/CMakeFiles/transformer_engine.dir/fused_attn/fused_attn.cpp.o FAILED: common/CMakeFiles/transformer_engine.dir/fused_attn/fused_attn.cpp.o /usr/bin/c++ -Dtransformer_engine_EXPORTS -I/**/TransformerEngine/transformer_engine -I/**/TransformerEngine/transformer_engine/common/include -I/**/TransformerEngine/transformer_engine/common/../../3rdparty/cudnn-frontend/include -I/**/TransformerEngine/build/cmake/common/string_headers -isystem /usr/local/cuda-12.3/targets/x86_64-linux/include -O3 -DNDEBUG -std=gnu++1y -fPIC -MD -MT common/CMakeFiles/transformer_engine.dir/fused_attn/fused_attn.cpp.o -MF common/CMakeFiles/transformer_engine.dir/fused_attn/fused_attn.cpp.o.d -o common/CMakeFiles/transformer_engine.dir/fused_attn/fused_attn.cpp.o -c /**/TransformerEngine/transformer_engine/common/fused_attn/fused_attn.cpp In file included from /**/TransformerEngine/transformer_engine/common/../../3rdparty/cudnn-frontend/include/cudnn_frontend_ConvDesc.h:32:0, from /**/TransformerEngine/transformer_engine/common/../../3rdparty/cudnn-frontend/include/cudnn_frontend.h:102, from /**/TransformerEngine/transformer_engine/common/fused_attn/utils.h:14, from /**/TransformerEngine/transformer_engine/common/fused_attn/fused_attn.cpp:9: /**/TransformerEngine/transformer_engine/common/../../3rdparty/cudnn-frontend/include/cudnn_frontend_utils.h:25:20: fatal error: optional: No such file or directory #include <optional> ^

[timmoon10]

Strange, filesystem and optional are both in the C++17 standard library and we explicitly specify C++17 support:

TransformerEngine/transformer_engine/CMakeLists.txt

Line 11 in 868c7d3

set(CMAKE_CXX_STANDARD 17)

However, I see C++14 flags (-std=gnu++1y) in your build logs. I wonder if CMake is misconfiguring something.

[ziyang-arch]

Yes. In the TransformerEngine/transformer_engine/CMakeLists.txt there are only set(CMAKE_CXX_STANDARD 17), and in the TransformerEngine/build/cmake/build.ninja I noticed for some files the flag becomes -std=gnu++1y and others are still c++17.
But I cannot find what caused the use of -std=gnu++1y in ninja.

[timmoon10]

If you are still debugging this, it may be helpful to pass the --verbose flag to pip install. This should print out the CMake build logs, which may give us clues why it's building with C++14 instead of C++17.

[ziyang-arch]

Thank you. I have solved this problem with switching from cmake 3.28 to 3.29 and re-cloning the latest repo.