Looks good. Something I'm wondering is how we go from the query in a scan's rules to embeddings that we're using to do VSS on redis.

Good question:

The idea is that targets which need to generate embeddings (Redis & Pinecone) will convert the incoming text to embeddings by calling the new embedding_create route directly. Usually when calling views directly, you're doing it from another view. Because we're doing it from a task, things get a little.... weird. While we do have the user object, constructing an HttpRequest object isn't very straightforward (you need the users session, and other stuff that is generally set by the middleware).

In order to get around from needing all that stuff, I'm going to refactor out all of the logic in embedding.views:create() into a library function that can be called directly by the task AND the view.

Example:

from embedding.lib import create_embedding

# Generate embedding from within the provider
embedding_data = create_embedding(text, model, service, user)