Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SQLite Hybrid Search and BM25 for local vector search #5073

Open
franciscojavierarceo opened this issue Feb 19, 2025 · 0 comments
Open
Labels
kind/feature New feature or request

Comments

@franciscojavierarceo
Copy link
Member

franciscojavierarceo commented Feb 19, 2025

Is your feature request related to a problem? Please describe.
See this example: https://github.com/liamca/sqlite-hybrid-search/tree/main and the sqlite docs: https://www.sqlite.org/fts5.html

This should be complemented with the SQLite-vec implementation.

Describe the solution you'd like

document_embeddings = FeatureView(
    name="embedded_documents",
    entities=[item, author],
    schema=[
        Field(
            name="vector",
            dtype=Array(Float32),
            # Look how easy it is to enable RAG!
            vector_index=True,
            vector_search_metric="COSINE",
        ),
        Field(name="item_id", dtype=Int64),
        Field(name="author_id", dtype=String),
        Field(name="created_timestamp", dtype=UnixTimestamp),
        Field(name="sentence_chunks", dtype=String),
        Field(name="event_timestamp", dtype=UnixTimestamp),
    ],
    source=rag_documents_source,
    ttl=timedelta(hours=24),
)

Somewhere in the FeatureView we should allow the search to be declared explicitly.

Describe alternatives you've considered
TBD

Additional context
Add any other context or screenshots about the feature request here.

@franciscojavierarceo franciscojavierarceo added the kind/feature New feature or request label Feb 19, 2025
@franciscojavierarceo franciscojavierarceo changed the title Add SQLite Hybrid Search for local vector search Add SQLite Hybrid Search and BM25 for local vector search Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant