-
Notifications
You must be signed in to change notification settings - Fork 9.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug: Kompute exits before loading model when offloading to GPU #8932
Comments
I get the same type of error (sometimes bus error, sometimes seg fault) when offloading to GPU on M2 Max. Doesn't happen every time though and doesn't happen running CPU only inference. |
For anyone having the same issue: |
I get that same vulkaninfo message with nomic-ai's fork (latest version) on Android 12 and a vulkan compliant gpu |
$ /data/local/tmp/llama-cli -m /sdcard/models/gg.gguf -ngl 999 --prompt "she once told me in bed" system_info: n_threads = 4 (n_threads_batch = 4) / 4 | AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 1 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | sampler seed: 1027837595 she once told me in bed, and I would not dream of sleeping with anyone else, least of all an ex. I had (debug log level) |
What happened?
I wanted to use the Kompute version to run on my GPU (Radeon RX570 4G) but whenever i use the
-ngl
argument to offload to GPU,llama-cli
silently exits before loading the model. When I tried to run the exact same command in MSYS2 mingw environment, I got same result (same log output) +Segmentation fault
message, so I assumed thats whats happening.The same model runs fine on my GPU with GPT4All (which, from what I understood, uses same backend).
Output of
vulkaninfo --summary
:here
Name and Version
What operating system are you seeing the problem on?
Windows
Relevant log output
The text was updated successfully, but these errors were encountered: