How is anyone affording this?

[a2jc4life]

I love the looks of this, but honestly, I'm kind of terrified to even try it, given that it seems we have to guess how much it will cost, and find out later. The chatbot says a token can be anywhere from approximately a character to approximately a word. My vault contains 4,069,804 words, or roughly 28,500,000 characters. And the current GPT costs $0.06 per thousand tokens, which is a lot more than the previous $0.0015. It looks like it would cost me $100s -- potentially $1000s since, again, we're guessing here at the number of tokens required -- just for initial processing.

By the estimation on this page: https://openai.com/pricing of 1000 tokens being equivalent to roughly 750 words, my vault would be about 5,426 sets of 1000 tokens. That's about $325. At a token per character, it would be more like ...$1700, if I did my math right.

Those are huge numbers, and that's huge variation.

How are people running this?

[brianpetro]

Hi @a2jc4life

I think you'd be surprised how low the actual cost is. Based on your numbers, initially embedding your entire vault should be ~$1. Maybe $2 max.

After that, the embedding cost is negligible.

For the chat models, you're charged on a per use basis. And for a single query, the most you're charged is the token limit for a single request per query. For the GPT-4, that will top out ~$0.25 per query, and for GPT-3.5-16K (my personal recommendation) it will top out at $0.05 per query. And GPT-3.5-4K would only be pennies per query.

I encourage you to give it a try and report back on your findings for anyone else that might be concerned about the cost.

Thanks for your interest in Smart Connections

🌴

[a2jc4life]

I'm kind of afraid to "just try it" since I won't know the cost until afterward. But setting the usage limits low beforehand is a good idea. I'm willing to just try it out if I can guarantee myself I'm not at risk of unexpectedly ending up with a $400 charge!

Just to be clear, on an unrelated point, it won't send excluded notes from excluded folders, right?

[quelsemme]

@a2jc4life If you're worried, put a limit on how much the OpenAI API will allow you to spend.

As @brianpetro suggested, and as have others have done, I just ran a test on reprocessing my vault. 506,497 words.
Processing embeddings using GPT-4 cost me $0.35

[jwhco]

@a2jc4life Start with a local model. You'll get a lot out of BGE-tiny and BGE-small -- I get good results on my 27k note vault. As you get familiar with Smart Connections, your comfort level will come up.

[arminta7]

Just as a point of reference:

My largest and main vault is over 20,000 files and 50,000,000 words. I use this plugin in that vault and have re-embedded the entire thing many times. I also use it on other smaller vaults of mine.

I also use plugins like text generator frequently. I do some of my own development using the OpenAI, even at times doing embedding processes I would have to leave overnight because they were quite large. I also use the API through other tools and apps. I almost always use GPT4.

Even with all of that, and considering that most of my heavier development took place when the costs were somewhat higher, my total costs have only been $350 since Nov 2022. Most months my cost is less than $10.

I hope that helps.

[jwhco]

If you use text generators and OpenAI to create content in your vault, you'll get more hallucinations from any AI model. There is a pattern in most AI responses that tends to work like photocopying something too many times. I'm not sure if there is a word for this yet.

[redington7]

These are my concerns as well and just starting to use chatGPT for more than entertainment. I'm assuming subscribing to ChatGPT Plus does not work because it uses an API? I'm also assuming you could set a soft $ limit to see if you wanted to continue and a hard $ limit to stop the process if it was getting too expensive knowing you might lose the money?!?

[brianpetro]

subscribing to ChatGPT Plus does not work

Correct, the API is a separate cost.

you could set a soft $ limit

Correct, OpenAI provides options to set limits in your API account.

🌴

[a2jc4life]

Update: I can't begin to fathom how the math works, but y'all are right. Unless it has a lot of vault still to process -- which it doesn't look like it does? -- it cost about $1.42 to initialize.

Based on what I've been doing with it using the chat, it's about 5 cents to process a moderately-sized note in a similar manner to what I was doing in OpenAI directly. That's a mixed-bag, though. To do it directly at is free. To do it within the plugin obviously saves having to go to a separate site. But I also found it was noticeably slower than using the OpenAI site, so...it depends, I guess, on what's most important to the particular user.

[cloudcathedral]

I believe that's because when you load the contents of your vault you're creating Embeddings, which is separate from chat/text generation, and Embedding is significantly cheaper - $0.0001 / 1K tokens

[quelsemme]

ChatGPT directly is just chatting with their LLM with no context from your notes. Of course, you could paste a note in, if you just want to ask a question about a note, but the real power of Smart Connections is the connections between notes through vectors/embeddings.

[psenger]

Is there a reason why this cant be modified to include a local AI?

I have Ollama installed and I use the API to get things resolved locally, all day. Obviously even on a maxed out M2 Pro, it is not as fast as OpenAI but its cheaper especially if you have a large vault.

[brianpetro]

No reason

🌴

[psenger]

@brianpetro I would be willing to do this, as I have the PLENTY of real world JavaScript and experience, plus Im working on a AI product right now.

However, my experience with working on OpenSource is "Lacking". I don't see a Contributing.MD. Do you have any guidelines, and are you willing to work with me as Im a Noob to this process? Would really enjoy adding this.

[brianpetro]

@psenger I appreciate that. Though, I think it'll be worth waiting until version 2, which will arrive shortly. V2 will be more amenable to additional models. Not to mention V1 isn't really fit for contributions because it was sort of a rough draft, hence the lack of docs for contributions.

🌴

[RazyRo]

@brianpetro I would be willing to do this, as I have the PLENTY of real world JavaScript and experience, plus Im working on a AI product right now.

However, my experience with working on OpenSource is "Lacking". I don't see a Contributing.MD. Do you have any guidelines, and are you willing to work with me as Im a Noob to this process? I would enjoy adding this.

Please any follow-up? I installed this plugin with much excitement to find out, I have connected to "Smart connect" which is an app to connect to Openai. how can I connect to Ollama?