Unable to run llama.cpp on A770, missing .so file

[KczBen]

Hello,

I have a fresh install of Ubuntu Server 24.04 on my machine. After following the guides at

https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quickstart/install_linux_gpu.md
https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md

I cannot get llama.cpp running, I get an error for a missing .so file:

./llama-cli: error while loading shared libraries: libggml-cpu.so: cannot open shared object file: No such file or directory

These are the steps I took to install the drivers:

# Adding apt repo
sudo apt-get install -y gpg-agent wget
wget -qO - https://repositories.intel.com/gpu/intel-graphics.key | \
  sudo gpg --dearmor --output /usr/share/keyrings/intel-graphics.gpg
echo "deb [arch=amd64,i386 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/gpu/ubuntu jammy client" | \
  sudo tee /etc/apt/sources.list.d/intel-gpu-jammy.list
sudo apt-get update

# Installing Mesa
sudo apt-get install -y udev \
  intel-opencl-icd intel-level-zero-gpu level-zero \
  intel-media-va-driver-non-free libmfx1 libmfxgen1 libvpl2 \
  libegl-mesa0 libegl1-mesa libegl1-mesa-dev libgbm1 libgl1-mesa-dev libgl1-mesa-dri \
  libglapi-mesa libgles2-mesa-dev libglx-mesa0 libigdgmm12 libxatracker2 mesa-va-drivers \
  mesa-vdpau-drivers mesa-vulkan-drivers va-driver-all vainfo

sudo reboot

Installing oneAPI:

# Creating group
sudo gpasswd -a ${USER} render
newgrp render

# Adding apt repo for oneAPI
wget -O- https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB | \
  gpg --dearmor | sudo tee /usr/share/keyrings/oneapi-archive-keyring.gpg > /dev/null
  
echo "deb [signed-by=/usr/share/keyrings/oneapi-archive-keyring.gpg] https://apt.repos.intel.com/oneapi all main" | \
  sudo tee /etc/apt/sources.list.d/oneAPI.list
  
sudo apt update

# Installing oneAPI
sudo apt install intel-oneapi-common-vars=2024.0.0-49406 \
  intel-oneapi-common-oneapi-vars=2024.0.0-49406 \
  intel-oneapi-diagnostics-utility=2024.0.0-49093 \
  intel-oneapi-compiler-dpcpp-cpp=2024.0.2-49895 \
  intel-oneapi-dpcpp-ct=2024.0.0-49381 \
  intel-oneapi-mkl=2024.0.0-49656 \
  intel-oneapi-mkl-devel=2024.0.0-49656 \
  intel-oneapi-mpi=2021.11.0-49493 \
  intel-oneapi-mpi-devel=2021.11.0-49493 \
  intel-oneapi-dal=2024.0.1-25 \
  intel-oneapi-dal-devel=2024.0.1-25 \
  intel-oneapi-ippcp=2021.9.1-5 \
  intel-oneapi-ippcp-devel=2021.9.1-5 \
  intel-oneapi-ipp=2021.10.1-13 \
  intel-oneapi-ipp-devel=2021.10.1-13 \
  intel-oneapi-tlt=2024.0.0-352 \
  intel-oneapi-ccl=2021.11.2-5 \
  intel-oneapi-ccl-devel=2021.11.2-5 \
  intel-oneapi-dnnl-devel=2024.0.0-49521 \
  intel-oneapi-dnnl=2024.0.0-49521 \
  intel-oneapi-tcm-1.0=1.0.0-435

sudo reboot

With the above setup, I was able to run a modified version of the demo (it seems the demo listed on the page is broken?)

Installing llama.cpp:

conda create -n llm-cpp python=3.11
conda activate llm-cpp
pip install --pre --upgrade ipex-llm[cpp]

mkdir llama-cpp
cd llama-cpp

init-llama-cpp

wget https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF/resolve/main/mistral-7b-instruct-v0.1.Q4_K_M.gguf

Attempting to run the llama.cpp demo:

source /opt/intel/oneapi/setvars.sh
export SYCL_CACHE_PERSISTENT=1
./llama-cli -m mistral-7b-instruct-v0.1.Q4_K_M.gguf -n 32 --prompt "Once upon a time, there existed a little girl who liked to have adventures. She wanted to go to places and meet new people, and have fun" -c 1024 -t 8 -e -ngl 99 --color
./llama-cli: error while loading shared libraries: libggml-cpu.so: cannot open shared object file: No such file or directory

I'm somewhat confused as to what could've gone wrong. Did I miss a step, or do I have the wrong version of some package? I also get a similar missing .so file error when trying to run ollama.

[sgwhat]

Hi @KczBen, You may install ollama through pip install ipex-llm[cpp]==2.2.0b20250204.

We will fix this issue today, and you may also try the latest version tmr.

[KczBen]

Thank you, that works! Now I have Ollama running, though I'm also encountering the same issue as #12761 with deepseek-r1:14b

[sgwhat]

Thank you, that works! Now I have Ollama running, though I'm also encountering the same issue as #12761 with deepseek-r1:14b

Could you pls provide some detailed logs? And in which round of conversation did the error begin to occur?

[sgwhat]

Hi @KczBen , for deepseek-r1 garbage output issue, you may refer to my response in #12761 (comment).

[KczBen]

I always got the issue on the second response from r1:14b. Using today's ollama with IPEX-LLM and oneAPI 2025, I have gotten 5 coherent messages so far in the same chat without specifying the longer context. I'm using the same prompts I did when I was getting the garbage outputs before.

I have oneAPI 2025 installed now as I was experimenting with upstream llama.cpp with SYCL, which requires 2025 to compile.

Is installing two separate versions of oneAPI supported? If so, I could try installing 2024.0 and 2025.0 side by side and see if that's affecting the behaviour. I have 2025 installed through apt.

[sgwhat]

Hi @KczBen, yes you can install two versions simultaneously.