ollama AssertionError for cli/python

[Mikkicon]

I followed a zero_to_hero_guide and am facing this issue for

llama_models==0.0.54
llama_stack==0.0.54
llama_stack_client==0.0.54
MODEL_NAME = "meta-llama/Llama-3.2-3B-Instruct"
OLLAMA_INFERENCE_MODEL = "llama3.2:3b-instruct-fp16"

llama-stack-client

$ llama-stack-client --endpoint http://localhost:5001 inference chat-completion --message "hello, what model are you?"

Failed to inference chat-completion                                                          
Error Type: InternalServerError                                                              
Details: Error code: 500 - {'detail': 'Internal server error: An unexpected error occurred.'}

ollama docker

  File "/usr/local/lib/python3.10/site-packages/llama_stack/providers/remote/inference/ollama/ollama.py", line 257, in _nonstream_chat_completion
    assert isinstance(r, dict)
AssertionError

[ashwinb]

cc @init27 @heyjustinai just fyi if you see this more in the wild, please report. this seems odd and seems like we had tested this path a bunch!

[Mikkicon]

@ashwinb FYI it works for this endpoint, so it is possibly an Ollama issue

LLAMA_STACK_API_TOGETHER_URL="https://llama-stack.together.ai"

[ashwinb]

@Mikkicon can you tell me

• what machine you are running on? Is it a server? a laptop? Mac? Showing output of uname -a would be useful
• what is your exact command for running the llama stack ollama docker?

[mukundsayeeganesh]

I am facing the same error

This is my machine info, I am running in github codespace -
Linux codespaces-74fe34 6.5.0-1025-azure #26~22.04.1-Ubuntu SMP Thu Jul 11 22:33:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Command for running the docker
docker run \ -it \ --add-host=host.docker.internal:host-gateway \ -p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \ -v ~/.llama:/root/.llama \ llamastack/distribution-ollama \ --port $LLAMA_STACK_PORT \ --env INFERENCE_MODEL=$INFERENCE_MODEL \ --env OLLAMA_URL=https://animated-waffle-9794x5jjwgqg295wv-11434.app.github.dev