Swap chat model provider at runtime

[ikwattro]

More of a question than an issue first as nothing is really mentioned in the docs.

Version : 0.24.0.CR1

With the following settings :

# langchain4j
quarkus.langchain4j.log-requests=true
quarkus.langchain4j.log-responses=true
quarkus.langchain4j.chat-memory.memory-window.max-messages=20

# openai
quarkus.langchain4j.m1.chat-model.provider=${M1_CHAT_MODEL_PROVIDER:ollama}
quarkus.langchain4j.openai.m1.api-key=${OPENAI_API_KEY:''}

# ollama
quarkus.langchain4j.ollama.base-url=${OLLAMA_BASE_URL:http://localhost:11434}

I was hoping to be able to switch between ollama and openai with the same build depending if I'm running locally or in deployed environment, but it seems that even when then M1_CHAT_MODEL_PROVIDER=openai variable is passed to the container ( as in docker container ), the app continues trying to use ollama.

Did I miss something or is the model provider constrained to be set at build time ? If it is the case, what is the recommended way to approach this ?

[geoand]

The model to use is indeed constrained to build time

[ikwattro]

Thanks @geoand , what is the recommendation on how to approach this ? Can I somehow register all of them with different names and choose one based on another config property ?

[geoand]

It depends on what exactly you want to achieve.

I believe @maxandersen had some example were was doing something similar

[edeandrea]

I've achieved this several times using several different approaches. With any of these approaches though it is still a build-time switch, but there aren't any code/config changes.

1. Use maven profiles combined with Quarkus profiles and have the dependencies in the profiles. I do this in https://github.com/edeandrea/non-deterministic-no-problem/blob/main/pom.xml, https://github.com/edeandrea/non-deterministic-no-problem/blob/main/src/main/resources/application.yml#L103, and https://github.com/edeandrea/non-deterministic-no-problem/blob/main/src/main/resources/application.yml#L121, as well as in this app (https://github.com/rh-rad-ai-roadshow/parasol-insurance/blob/main/app)

2. Have multiple extensions in your pom and use quarkus profiles to switch between them:

https://github.com/quarkusio/quarkus-super-heroes/blob/0464b924645f84a05c7bbd494e72f5645bc2231e/rest-narration/src/main/resources/application.properties#L4-L84