Dynamic Prompt Change according to model name

[Arshil-Akkala]

Is it possible to dynamically load respective prompts/ prompt templates based on the model_name selected on UI? Will it include a UI change or is there any provision available for the same?

[mmabrouk]

Hi @Arshil-Akkala,

Can you provide more information about the use case / the problem you are trying to solve / your workflow here?

Technically, that would be possible yes. You could write your code so that when a certain model is selected in the UI, (model=="somemodel") then you would load some prompt or post-process the prompt value before passing it to the model.

But I'd be able to provide better help if you could explain more the use case.

Best,
Mahmoud

[Arshil-Akkala]

Let us say, I am having some models in the dropdown, each having their own default prompts. So, my intention is to dynamically load respective prompts according to the model selected.

[mmabrouk]

Yes that would work. Here is how the code could look like:

import agenta as ag
import litellm


litellm.drop_params = True
litellm.callbacks = [ag.callbacks.litellm_handler()]

supported_models = {
    "Mistral AI": [
        "mistral/mistral-tiny",
        "mistral/mistral-small",
        "mistral/mistral-medium",
        "mistral/mistral-large-latest",
    ],
    "Open AI": [
        "gpt-3.5-turbo-1106",
        "gpt-3.5-turbo",
        "gpt-4",
        "gpt-4o",
        "gpt-4-1106-preview",
    ], }

prompts = {
    "user_prompt": """What is the capital of {country}?""",
}


ag.init()
ag.config.default(
    temperature=ag.FloatParam(default=1, minval=0.0, maxval=2.0),
    model=ag.GroupedMultipleChoiceParam(
        default="gpt-3.5-turbo", choices=supported_models
    ),
    max_tokens=ag.IntParam(-1, -1, 4000),
    prompt_user=ag.TextParam(prompts["user_prompt"]),
    top_p=ag.FloatParam(1),
    frequence_penalty=ag.FloatParam(default=0.0, minval=-2.0, maxval=2.0),
    presence_penalty=ag.FloatParam(default=0.0, minval=-2.0, maxval=2.0),
)


@ag.entrypoint
@ag.instrument()
async def generate(
    inputs: ag.DictInput = ag.DictInput(default_keys=["country"]),
):
    try:
        prompt_user = ag.config.prompt_user.format(**inputs)
    except Exception as e:
        prompt_user = ag.config.prompt_user

    max_tokens = ag.config.max_tokens if ag.config.max_tokens != -1 else None

    completion_params = {}

    system_prompt_gpt = "WE ONLY ADD THIS IF IT IS A GPT MODEL"
    system_prompt_mistral = "WE ONLY ADD THIS IF IT IS A GPT MODEL"
    prompt_system = system_prompt_gpt if ag.config.model.startswith(
        "gpt") else system_prompt_mistral

    response = await litellm.acompletion(
        **{
            "model": ag.config.model,
            "messages": [
                {"content": prompt_system, "role": "system"},
                {"content": prompt_user, "role": "user"},
            ],
            "temperature": ag.config.temperature,
            "max_tokens": max_tokens,
            "top_p": ag.config.top_p,
            **completion_params,
        }
    )
    token_usage = response.usage.dict()
    return {
        "message": response.choices[0].message.content,
        "usage": token_usage,
        "cost": litellm.cost_calculator.completion_cost(
            completion_response=response, model=ag.config.model
        ),
    }

[Arshil-Akkala]

So, we can handle this entirely from the backend (custom app) right?

[mmabrouk]

Yes, exactly. You can have any custom logic in your LLM app, that could be loading a prompt dynamically for a certain model. The prompt could even be loaded from another system.

prompt_system = load_my_gpt_prompt() if ag.config.model.startswith(
        "gpt") else system_prompt_mistral

[Arshil-Akkala]

That's amazing! Thanks, will try it and will reach out

[Arshil-Akkala]

Umm there is still one thing bugging me, in the backend we are making changes to handle that. But how will the UI know the change being made, I mean the UI also has to render the prompt according to the model change right.?

[mmabrouk]

The UI would not be rendering the prompt (unless you are putting it in the multiple choice). So, the prompt would be used, but you will not see it in the playground. You will see it in observability though.

[mmabrouk]

Can you please explain more about the use case. I don't understand why you would want to have the prompts hard coded for different models, and not create different variants with a prompt/model pairs?

[Arshil-Akkala]

why you would want to have the prompts hard coded for different models, and not create different variants with a prompt/model pairs?, to answer this, let us say we have a, b, c models and to create variants out of it, I need to have their base versions right. So that is what my intention is.

[mmabrouk]

I am not sure I understood..

You have the option to create a custom application that provides as a config a multi-choice for models A,B and C and the prompt. You can then create three variants, one for model A with prompt_model_A, one for model B with prompt_model_B, and one for model C with prompt_model_C. And save each. Why would that not work?

[Arshil-Akkala]

I tried that, it doesn't seem to be quite that way, or may be my approach is incorrect..

`
litellm.drop_params = True
litellm.callbacks = [ag.callbacks.litellm_handler]

CUSTOM_LLMS = [
LLM_MISTRAL_NAME,
LLM_ZEPHYR_NAME
]

PROMPT_TEMPLATES = {
LLM_MISTRAL_NAME: mixtral_prompt,
LLM_ZEPHYR_NAME: zephyr_prompt,
}

DEFAULT_KEYS = {
LLM_MISTRAL_NAME: ["input"],
LLM_ZEPHYR_NAME: ["sys_msg", "user_prompt"]
}

ag.init()
ag.config.default(
prompt_template=ag.TextParam(PROMPT_TEMPLATES[LLM_MISTRAL_NAME]),
temperature=ag.FloatParam(LLM_TEMPERATURE),
model_name=ag.MultipleChoiceParam(LLM_MISTRAL_NAME, CUSTOM_LLMS),
top_p=ag.FloatParam(LLM_TOP_P),
top_k=ag.IntParam(0, 0, 10),
repetition_penalty=ag.FloatParam(LLM_REPEATITION_PENALTY),
max_tokens=ag.IntParam(LLM_MAX_TOKENS, -1, LLM_MAX_TOKENS),
)

@ag.entrypoint
@ag.instrument()
def generate(inputs: ag.DictInput = ag.DictInput(default_keys=DEFAULT_KEYS[ag.config.model_name])):
selected_model = ag.config.model_name
prompt_template_str = PROMPT_TEMPLATES[selected_model]

final_prompt = prompt_template_str.format(**inputs)
prompt = PromptTemplate(
    input_variables=[], template=final_prompt
)

llm = some_llm_manager(
    repo_id=selected_model, 
    endpoint_url=MODEL_HOST,
    task=LLM_TASK,
    model_kwargs={
        "temperature": ag.config.temperature,
        "top_p": ag.config.top_p,
        "top_k": ag.config.top_k,
        "repetition_penalty": ag.config.repetition_penalty,
        "max_new_tokens": ag.config.max_tokens
    }, 
    verbose=False
)

chain = LLMChain(llm=llm, prompt=prompt)
output = chain.run(inputs)

return output

`
Perhaps you can help me in this.