Unable to install llama-cpp-python Package in Python - Wheel Building Process gets Stuck

[Illanser]

I’m trying to install the llama-cpp-python package in Python, but I’m encountering an issue where the wheel building process gets stuck. Here’s the command I’m using to install the package:

pip3 install llama-cpp-python

The process gets stuck at this step:

Building wheel for llama-cpp-python (pyproject.toml)...

I’ve tried increasing the verbosity of the output with the -v option, but it didn’t provide any additional useful information. I’ve also tried installing an older version of the package, but I encountered the same issue.

I’ve confirmed that I have the latest versions of pip, setuptools, and wheel installed. My system resources (CPU, memory) also seem to be sufficient.

Here’s what I’ve tried so far:

1. Increasing verbosity: Running pip install -v llama-cpp-python to get more details about what’s happening during the installation process.
2. Installing an older package version: This was suggested as a potential solution, but it didn’t resolve the issue.
3. Upgrading pip, setuptools, and wheel: Ensuring that I have the latest versions of these tools.
   Despite these efforts, I’m still unable to install the llama-cpp-python package. Has anyone encountered this issue before or have any suggestions for resolving it?

My Linux Version is: Ubuntu 22.04.3 LTS

Any help would be greatly appreciated! I really appreciate any help you can provide.

[watanka]

you need llama cpp compiler to pip install.
Have you tried
apt-get install build-essential -y and then pip install llama-cpp-python?

[etdr]

I don't know if this is the same issue, but I'm getting a CMake error that is preventing me from building:

CMake Error at /usr/share/cmake/Modules/CMakeDetermineCompilerId.cmake:753 (message):
        Compiling the CUDA compiler identification source file
        "CMakeCUDACompilerId.cu" failed.

Here's a potentially relevant part of the output:

#$ cudafe++ --c++17 --gnu_version=130201 --display_error_number
--orig_src_file_name "CMakeCUDACompilerId.cu" --orig_src_path_name
"/tmp/tmp1bbhzucj/build/CMakeFiles/3.27.7/CompilerIdCUDA/CMakeCUDACompilerId.cu"
--allow_managed --m64 --parse_templates --gen_c_file_name
"tmp/CMakeCUDACompilerId.cudafe1.cpp" --stub_file_name
"CMakeCUDACompilerId.cudafe1.stub.c" --gen_module_id_file
--module_id_file_name "tmp/CMakeCUDACompilerId.module_id"
"tmp/CMakeCUDACompilerId.cpp4.ii"

/usr/include/bits/floatn.h(86): error: invalid combination of type
specifiers

  typedef __float128 _Float128;
                     ^



/usr/include/bits/floatn-common.h(214): error: invalid combination of type
specifiers

  typedef float _Float32;
                ^



/usr/include/bits/floatn-common.h(251): error: invalid combination of type
specifiers

  typedef double _Float64;
                 ^



/usr/include/bits/floatn-common.h(268): error: invalid combination of type
specifiers

  typedef double _Float32x;
                 ^



/usr/include/bits/floatn-common.h(285): error: invalid combination of type
specifiers

  typedef long double _Float64x;
                      ^



5 errors detected in the compilation of "CMakeCUDACompilerId.cu".

# --error 0x2 --

Call Stack (most recent call first):
  /usr/share/cmake/Modules/CMakeDetermineCompilerId.cmake:8 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
  /usr/share/cmake/Modules/CMakeDetermineCompilerId.cmake:53 (__determine_compiler_id_test)
  /usr/share/cmake/Modules/CMakeDetermineCUDACompiler.cmake:307 (CMAKE_DETERMINE_COMPILER_ID)
  vendor/llama.cpp/CMakeLists.txt:290 (enable_language)

I definitely have a compiler installed. I'm on CUDA 12.3, if that helps.

[watanka]

it seems like your cuda version and gcc version are not compatible. Not sure but try match their version?

[JettScythe]

I was able to get around this with CUDAHOSTCXX=/usr/bin/g++-12

[arcolife]

btw it was OOM'ing and ate all of 128G RAM while building wheels.

When I tried via uv pip install llama-cpp-python it worked.