Impressive 12,112 tokens/sec TaylorAI/bge-micro-v2

[jwhco]

Smart Connections v2.1.99

Before the GPU update ... 5,500 tokens/sec TaylorAI/bge-micro-v2
After the GPU update ... 12,112 tokens/sec TaylorAI/bge-micro-v2

With other models, I've seen an increase, but I'm having a challenge with Chat going away.

With bge-micro-v2 everything works so I can get back to work. Embedding is blazing fast. Thanks!

[jwhco]

This is no longer the case V2.2.76. Chat results are horrible. I'm trying to figure out from the change log what happened. Only bge-micro-v2 works.

Many changes in code seem trivial and random, but it may be a lack of understanding on my part. I'm noticing a lot of configuration changes that undo previous settings.

What I'm trying to accomplish is a local LM to associate content, allow chat, and summarize content. I don't want GPT content introduced, unless I specifically ask for it.

Hallucinations include dumbing down notes I already have but contrary to what notes say. For example, if a note says, "A 401k is a bad investment" a suggestion to "save 3% of earnings into a 401k" is counter.

Starting a session is also taking much longer. I don't see OpenRouter in the settings anymore. I was going to try to roll back to a previous version, yet I'm in the middle of a move so I don't have the time.

I work without Smart Connection on my laptop. I had the lean coffee call on my schedule, but couldn't attend. There is a lot of value here. It's hard to tell if stability issues are my vault or the software.