Conversion error

[TMust77]

Please help me with this error running convert.py, I am running it with Cuda 12.1 and a Tesla P100:

Loading exllamav2_ext extension (JIT)...
Building C++/CUDA extension ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2% 0:00:38 -:--:--
Traceback (most recent call last):
File "C:\Users\timom\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils\cpp_extension.py", line 2096, in _run_ninja_build
subprocess.run(
File "C:\Users\timom\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "D:\Temp\Download\text-generation-webui-1.10.1\models\exllamav2\convert.py", line 1, in
import exllamav2.conversion.convert_exl2
File "D:\Temp\Download\text-generation-webui-1.10.1\models\exllamav2\exllamav2_init_.py", line 3, in
from exllamav2.model import ExLlamaV2
File "D:\Temp\Download\text-generation-webui-1.10.1\models\exllamav2\exllamav2\model.py", line 35, in
from exllamav2.config import ExLlamaV2Config
File "D:\Temp\Download\text-generation-webui-1.10.1\models\exllamav2\exllamav2\config.py", line 5, in
from exllamav2.fasttensors import STFile
File "D:\Temp\Download\text-generation-webui-1.10.1\models\exllamav2\exllamav2\fasttensors.py", line 6, in
from exllamav2.ext import exllamav2_ext as ext_c
File "D:\Temp\Download\text-generation-webui-1.10.1\models\exllamav2\exllamav2\ext.py", line 271, in
exllamav2_ext = load
File "C:\Users\timom\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils\cpp_extension.py", line 1306, in load
return jit_compile(
File "C:\Users\timom\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils\cpp_extension.py", line 1710, in jit_compile
write_ninja_file_and_build_library(
File "C:\Users\timom\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils\cpp_extension.py", line 1823, in write_ninja_file_and_build_library
run_ninja_build(
File "C:\Users\timom\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils\cpp_extension.py", line 2112, in run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'exllamav2_ext': [1/46] D:\Program Files (x86)\Cuda\bin\nvcc --generate-dependencies-with-compile --dependency-output h_add.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -ID:\Temp\Download\text-generation-webui-1.10.1\models\exllamav2\exllamav2\exllamav2_ext -IC:\Users\timom\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\include -IC:\Users\timom\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\timom\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\include\TH -IC:\Users\timom\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\include\THC "-ID:\Program Files (x86)\Cuda\include" -IC:\Users\timom\AppData\Local\Programs\Python\Python310\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_60,code=compute_60 -gencode=arch=compute_60,code=sm_60 -std=c++17 -lineinfo -O3 -c D:\Temp\Download\text-generation-webui-1.10.1\models\exllamav2\exllamav2\exllamav2_ext\cuda\h_add.cu -o h_add.cuda.o
FAILED: h_add.cuda.o
D:\Program Files (x86)\Cuda\bin\nvcc --generate-dependencies-with-compile --dependency-output h_add.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -ID:\Temp\Download\text-generation-webui-1.10.1\models\exllamav2\exllamav2\exllamav2_ext -IC:\Users\timom\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\include -IC:\Users\timom\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\timom\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\include\TH -IC:\Users\timom\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\include\THC "-ID:\Program Files (x86)\Cuda\include" -IC:\Users\timom\AppData\Local\Programs\Python\Python310\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compninja: fatal: ReadFile: The handle is invalid.

ute_60,code=compute_60 -gencode=arch=compute_60,code=sm_60 -std=c++17 -lineinfo -O3 -c D:\Temp\Download\text-generation-webui-1.10.1\models\exllamav2\exllamav2\exllamav2_ext\cuda\h_add.cu -o h_add.cuda.o
CreateProcess failed: The system cannot find the file specified.

[turboderp]

It looks like you don't have ninja installed. pip install ninja should fix that. To build from source you'll also need the CUDA toolkit and VS Build Tools (or Visual Studio.) Alternatively you can install a prebuilt wheel to skip all that. For your setup I'm guessing it would be:

pip install https://github.com/turboderp/exllamav2/releases/download/v0.1.7/exllamav2-0.1.7+cu121.torch2.3.1-cp310-cp310-win_amd64.whl

Can't tell the Torch version from that stack trace, but there are more wheels here if you're not using 2.3.1.

[TMust77]

Pip install ninja reports:
Requirement already satisfied: ninja in c:\users\timom\appdata\local\programs\python\python310\lib\site-packages (1.11.1.1)

Will try the prebuilt wheel and report if that fixes it. Thanks!