Pre-Embedding Token Count Analysis

[SquidLord]

The short version:

It would be extremely convenient to be able to call this plug-in and have it do a breakdown of exactly how large your Vault happens to be in terms of tokens, which would give you some idea of how expensive it will be to ingest the whole thing for creating embeddings.

It's possible that I don't like surprises and I'm cheap. Or frugal. Either way.

In a similar vein, it might be quite useful to be able to only call for new embeddings to be generated from new/updated notes once or twice a day outside of normal work hours, which could save on the number of transactions and tokens transferred for highly dynamic notes.

Thanks much for a fascinating tool.

[jpchauny]

Oh! You made me think about high dynamical notes… good analysis and good idea.

[SquidLord]

For people who use a single note as the central contact point for their daily operations or people who have central hubs that get updated multiple times a day, being able to control how often that goes to hit the API can be a matter of several dollars. That adds up.

I almost think that if we are going to be breaking down content into blocks smaller than a file anyway, we should really only be building embeddings for the new blocks rather than breaking the whole thing down into blocks and submitting each of those blocks again to be turned into embeddings.

Which brings to mind a different way to deal with notes which have a high rate of dynamic change:

• Break them down into blocks.
• Compare to the last time we broke that file down into blocks with a diff.
• Only build embeddings for blocks which have changed.
• Proceed as usual.

In that case, at least you won't be submitting the same information to be broken down repeatedly, but it's also a situation in which it's probably better to stop using a single JSON file and switch over to some sort of proper vector database so that you could associate blocks with files and know when things have changed with some sort of checksum alongside.

Probably a good idea to start looking at, anyway.

[brianpetro]

Hi @SquidLord

Compare to the last time we broke that file down into blocks with a diff.
Only build embeddings for blocks which have changed.

Already doing this. You can turn on additional logging in the settings to see in the console log.

but it's also a situation in which it's probably better to stop using a single JSON file and switch over to some sort of proper vector database so that you could associate blocks with files and know when things have changed with some sort of checksum alongside.

Are you aware of some VectorDB that handles synchronization with checksums like you're suggesting? Otherwise see #197 (comment)

Thanks for your interest in Smart Connections 😊
Brian 🌴

[SquidLord]

You should be able to store checksums for blocks along with vectors and some other file info metadata using Weaviate. My most recent database experience is with more graph gunk, but pretty much any database is capable of holding a multitude of useful bits beside one another.

Injecting meta-info to the query can be very handy on some occasions (like telling the prompt to unify file references, etc).

[brianpetro]

There is a meta object with checksum (MD5 hashes) for each vector in the JSON file.