Gemma 4-31b from Model Gallery Broken.

[JohnGalt1717]

LocalAI version:
LocalAI v4.1.3 (fdc9f7b)

Environment, CPU architecture, OS, and Version:
Docker:
Host: linux linuxserver 6.8.0-111-generic #111-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 11 23:16:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

Describe the bug
Choosing the Gemma-4-31b model from the gallery doesn't work. Errors with: Error: failed to load model with internal loader: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF

To Reproduce
Choose it from the gallery download, go to chat, pick the model, and say "hi".

Expected behavior
Should work with the right template in for gemma4 for tool calling, etc.

Logs
Nothing shows up.

Additional context
The gemma models were all updated 3 days ago because of the major speed improvements google did so likely these are now broken in the gallery and need to be updated for the new models and force the correct version of llama.cpp or vllm to work.

[mithrandir42]

The error message does not not say much. Could be anything. It is the same I got in #9696, but only provided the backend traces. Backend traces would be useful here, too.
To get traces, navigate to settings - tracing. Then enable tracing and backend logging. Once you get the error you can click on the onto [View traces for details] and get directly to the traces.

[JohnGalt1717]

backend-logs-gemma-4-31b-it-2026-05-07.json

Here you go!

[mithrandir42]

From what I can see from the traces it looks like you ran out of graphics memory:

    "timestamp": "2026-05-07T18:56:04.128510127Z",
    "stream": "stderr",
    "text": "ggml_backend_cuda_buffer_type_alloc_buffer: allocating 10240.00 MiB on device 0: cudaMalloc failed: out of memory"

In model gallery does it say fits or may not fit?

Here some guys are discussing the high memory needs of the model with some ideas for optimization: https://www.reddit.com/r/LocalLLaMA/comments/1sb4chx/gemma4_31b_unslothgamma431bitgguf_udq4_k_xl/?tl=de

From information in the model gallery the Gemma 4-31b needs 19.5 GB of VRAM. Yo are using two graphics cards, each with 15949 MiB VRAM. So the model could only be used when split across both cards. I would suggest trying to get the model splitted across both cards:
https://www.reddit.com/r/LocalLLaMA/comments/1kvp4nq/what_are_the_restrictions_regarding_splitting/

This should be possible by adjusting model configuration: https://localai.io/advanced/model-configuration/. You might get the model to work trying this.

@mudler I think out of memory condition might be the most common error. To reduce the amount of bug reports imho it might be a good idea to propagate more error details to the frontend. So the user directly sees what the problem is. Is this feasible?

[JohnGalt1717]

The 24b version of the model from 2 weeks ago works fine and uses 87% of both cards with the same configuration actually using more memory.

The new version of the 24b with the same quantization fails. Here's the definition for the 24b that works fine.

backend: llama-cpp
function:
automatic_tool_parsing_fallback: true
grammar:
disable: true
known_usecases:
- chat
min_p: 0
mmproj: mmproj-gemma-4-26B-A4B-it-f16.gguf
name: gemma-4-26b-a4b-it
options:
- use_jinja:true
parameters:
model: gemma-4-26B-A4B-it-Q4_K_M.gguf
repeat_penalty: 1
temperature: 1
template:
use_tokenizer_template: true
top_k: 64
top_p: 0.95