Improve token cost calculation (+performance)

[Shackless]

SlickGPT uses gpt3-tokenizer which is one of the few libs I found that "just runs" in the browser. It's close enough but not perfect as the calculations in

slickgpt/src/misc/openai.ts

Line 69 in af5d5e7

export function estimateChatCost(chat: Chat): ChatCost {

are a bit "off" compared to the official OpenAI Tokenizer.

Another and bigger problem is performance. The gpt3-tokenizer has a huge payload and it's pretty slow. Other solutions use advanced stuff like Node Buffer structures and are much faster. The problem is that they don't run easily in the browser without Node.

Any ideas how to calculate the tokens per Chat (context, prompts, completions) more accurately and faster?

[rarestg]

The official Tokenizer cites this package: https://www.npmjs.com/package/gpt-3-encoder
... or is this one of the libs that doesn't "just run" in the browser?

[Shackless]

The official Tokenizer cites this package: https://www.npmjs.com/package/gpt-3-encoder ... or is this one of the libs that doesn't "just run" in the browser?

Yeah, unfortunately this one requires a full node environment to run. We COULD use Vercel functions for that but I don't think it's worth it and would rather leave the token calculation to the client.