Support for LMStudio

[srikanth235]

No description provided.

[ivucica]

While I won't have time to experiment and patch privy/lib/extension/src/ai/AIClient.ts, here's some info about a hypothetical case of provider.startsWith("openai") or provider.startsWith("lm-studio"):

2024-11-29 15:01:52  [INFO] [LM STUDIO SERVER] Supported endpoints:
2024-11-29 15:01:52  [INFO] [LM STUDIO SERVER] ->	GET  http://192.0.0.2:5001/v1/models
2024-11-29 15:01:52  [INFO] [LM STUDIO SERVER] ->	POST http://192.0.0.2:5001/v1/chat/completions
2024-11-29 15:01:52  [INFO] [LM STUDIO SERVER] ->	POST http://192.0.0.2:5001/v1/completions
2024-11-29 15:01:52  [INFO] [LM STUDIO SERVER] ->	POST http://192.0.0.2:5001/v1/embeddings

Docs are here: https://lmstudio.ai/docs/api/openai-api

OpenAI's client should be usable with LM Studio, per LM Studio's documentation:

import OpenAI from 'openai';

const client = new OpenAI({
+  baseUrl: "http://localhost:1234/v1"
});

They provide a Python example, presumably a Typescript/Javascript one is similar:

# Example: reuse your existing OpenAI setup
from openai import OpenAI

# Point to the local server
client = OpenAI(base_url="http://localhost:1234/v1", api_key="lm-studio")

completion = client.chat.completions.create(
  model="model-identifier",
  messages=[
    {"role": "system", "content": "Always answer in rhymes."},
    {"role": "user", "content": "Introduce yourself."}
  ],
  temperature=0.7,
)

print(completion.choices[0].message)

However, I have used /v1/chat/completions directly with fetch() to avoid a dependency for something that's reasonably simple to do without extra dependencies:

fetch(`http://${host}/v1/chat/completions`, {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json'
    },
    body: JSON.stringify(request)
}).then(response => {
    if (response.ok) {
        return response.json();
    }
    throw new Error('Failed to get response: ' + response.statusText);
}).then(data => {
    console.log('Response:', data);
    let raw_json = data.choices[0].message.content;
    // we could decode the response, but we could also just write it out:
    console.log(raw_json);
}).catch(error => {
    console.error('Error:', error);
});

I've built the request like so:

let request = {
    messages: [
        { role: 'system', content: 'You are a helpful assistant and will return a response in JSON.' }, // note: this is because I was supplying a JSON schema.
        { role: 'assistant', content: prompt }
    ],
    temperature: 0.7,
    max_tokens: -1,
    stream: false,
};
if (model) {
    request.model = model; // model can be omitted if exactly one model is loaded in LM Studio GUI 
};
if (schema) { // we want to force the model to generate JSON according to a JSON schema
    request.response_format = {
        type: 'json_schema',
        json_schema: {
            name: 'some.name.for.schema.json',
            strict: true,
            schema: schema // deserialized JSON Schema object 
        },
    }
}

Given at least some endpoints are supported by OpenAI, this might be a common enough endpoint that it might unlock usability of more runtimes.

LM Studio is rather trivial to experiment with on Windows and Mac; I've had luck on AMD GPU on Windows and on Apple Silicon. I have not tried LM Studio on Linux at this time.

[ivucica]

Note that depending on what you want to do, this might even simplify the codebase given Ollama says they have OpenAI API compatibility nowadays:

https://ollama.com/blog/openai-compatibility