Vertex AI Hosted Llama Model Config.yaml

[faizan30]

Hi, I have hosted a llama2 model on vertex ai. I'm struggling to connect to it via litellm. Calling the endpoint directly works via requests. Here's the config.yaml I'm using:

model_list:

• model_name: vertex_ai/98563787278ff7
  litellm_params:
  - model: vertex_ai/98563787278ff7
  - vertex_project: "prod-312185"
  - vertex_location: "us-central1"
  - vertex_credentials: "/Users/faiz/helloworld-312185-a956eef1a2a8.json"

Kindly let me know if i'm missing something, any help would be appreciated :)

What i've already tried

1. https://docs.litellm.ai/docs/providers/custom
2. https://docs.litellm.ai/docs/providers/vertex#fine-tuned-models
3. https://docs.litellm.ai/docs/pass_through/vertex_ai