How to set up chat-ui with an openai endpoint?

[erickrf]

I'm trying to set up chat-ui using a remote endpoint that uses the openai API.

It is running on Kubernetes, and since we use a self-signed certificate, I also set up port forwarding, mapping localhost:8080 to the endpoint.

Long story short, I can send requests to the server via curl or Python code. For example, this works perfectly fine:

curl -H "content-type:application/json" http://localhost:8080/openai/v1/chat/completions -d '{"model": "llama3", "messages": [{"role": "user","content": "Where is Paris located?"}], "stream":false }'

But I cannot figure out how to set up the .env.local file correctly to use the server. I currently have the following:

MODELS=`[
  {
    "name": "Llama 3",
    "tokenizer": "meta-llama/Meta-Llama-3-8B",
    "preprompt": "You are a helpful AI assistant",
    "chatPromptTemplate": "<s>[INST] <<SYS>>\n{{preprompt}}\n<</SYS>>\n\n{{#each messages}}{{#ifUser}}{{content}} [/INST] {{/ifUser}}{{#ifAssistant}}{{content}} </s><s>[INST] {{/ifAssistant}}{{/each}}",
    "parameters": {
      "stop": ["<|end|>", "<|endoftext|>", "<|assistant|>"],
      "temperature": 0.7,
      "max_new_tokens": 1024,
      "truncate": 3071
    },
    "endpoints": [{
      "type": "openai",
      "baseURL": "http://localhost:8080/openai/"
    }],
  },
]`
MONGODB_URL=mongodb://localhost:27017

But whenever I start a conversation, chat-ui gets a 404 error from the server:

NotFoundError: 404 status code (no body)
    at APIError.generate (file:///Users/erick/projects/chat-ui/node_modules/openai/error.mjs:50:20)
    at OpenAI.makeStatusError (file:///Users/erick/projects/chat-ui/node_modules/openai/core.mjs:256:25)
    at OpenAI.makeRequest (file:///Users/erick/projects/chat-ui/node_modules/openai/core.mjs:299:30)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async eval (/Users/erick/projects/chat-ui/src/lib/server/endpoints/openai/endpointOai.ts:111:36)
    at async Module.generate (/Users/erick/projects/chat-ui/src/lib/server/textGeneration/generate.ts:8:30)
    at async textGenerationWithoutTitle (/Users/erick/projects/chat-ui/src/lib/server/textGeneration/index.ts:56:3) {
  status: 404,
  headers: {
    'content-length': '22',
    'content-type': 'application/json',
    date: 'Mon, 19 Aug 2024 14:34:00 GMT',
    server: 'uvicorn'
  },
  request_id: undefined,
  error: undefined,
  code: undefined,
  param: undefined,
  type: undefined
}
[16:34:00.877] ERROR (43095): 404 status code (no body)
    err: {
      "type": "NotFoundError",
      "message": "404 status code (no body)",
      "stack":
          Error: 404 status code (no body)
              at APIError.generate (file:///Users/erick/projects/chat-ui/node_modules/openai/error.mjs:50:20)
              at OpenAI.makeStatusError (file:///Users/erick/projects/chat-ui/node_modules/openai/core.mjs:256:25)
              at OpenAI.makeRequest (file:///Users/erick/projects/chat-ui/node_modules/openai/core.mjs:299:30)
              at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
              at async eval (/Users/erick/projects/chat-ui/src/lib/server/endpoints/openai/endpointOai.ts:111:36)
              at async Module.generateFromDefaultEndpoint (/Users/erick/projects/chat-ui/src/lib/server/generateFromDefaultEndpoint.ts:11:23)
              at async generateTitle (/Users/erick/projects/chat-ui/src/lib/server/textGeneration/title.ts:54:10)
              at async Module.generateTitleForConversation (/Users/erick/projects/chat-ui/src/lib/server/textGeneration/title.ts:17:19)
      "status": 404,
      "headers": {
        "content-length": "22",
        "content-type": "application/json",
        "date": "Mon, 19 Aug 2024 14:34:00 GMT",
        "server": "uvicorn"
      }
    }

This is similar but not quite the same error as #1311
How do I set it up?

[erickrf]

I finally found it out myself. The endpoint should be "http://localhost:8080/openai/v1". This was confusing, since the README says that chat-ui uses v1/chat/completions by default.