Description: Pinecone challenge Web UI searching Pinecone.io using hybrid search
Author: Kevin M. Butler
Date: Nov-2022
Version: 1.0
Purpose: This repository contains code to deploy a Python Flask application that is able to query Pinecone using vector search and more consicely, hybrid search to include dense and sparse vectors.
Introduction
Getting started
Resources
The Pinecone vector database makes it easy to build high-performance vector search applications. Developer-friendly, fully managed, and easily scalable without infrastructure hassles.
Vector databases have become very popular in semantic use-cases and now, Pinecone has introduced the ability to combine dense and sparse vectors together to perform hybrid search.
Hybrid search is the ability to use tokenized keywords (sparse vectors) and semantic representations (dense vectors) together to perform searches based on both lexical and semantic meanings. Traditionally, this would require two technologies, multiple queries, and post-processing to accomplish this task. With Pinecone hybrid search, you can now merge these technologies together in one fast, scalable, fully managed system.
This repository aims to demonstrate a web-crawl of pinecone.io, stored in Pinecone, and queried using a Python Flask application as the web UI. The web UI processes the query into sparse and dense vectors, then sends the query to Pinecone. The results are in json format and can be easily parsed by the web application.
The distinction of lexical versus semantic is controlled by the 'alpha' parameter which is passed in as a value between 0 and 1. A lower value closer to 0 is considered more lexical or keyword oriented. A higher value closer to 1 is more semantic and gives more weight to the meaning of the words rather than the more exact matching of the terms that lexical is suited for.
The ./resources directory has the files necessary to process the data and upload it to your Pinecone index.
The Jupyter notebook 'Pinecone_io_generate_embeddings_example.ipynb' is where you will run the code. Feel free to make a copy of this file and rename it to Pinecone_io_generate_embeddings.ipynb' It uses 'Pinecone_io_Webcrawl.json' as its data-source. To run this notebook, you will need to resolve any dependencies in your Python environment. You will also need your API key, environment and index name. If you do not have an API key, you can sign up for free here.
Before you beging working in the notebook, you will have to create a project in order to retrieve your API key. Once the project is created and you have your API, you can continue here.
If you need to create your index for the first time, you can use CURL
(modify YOUR_API_KEY, YOUR_ENVIRONMENT, and YOUR_INDEX_NAME below)
curl --location --request POST 'https://controller.YOUR_ENVIRONMENT.pinecone.io/databases' \
--header 'Api-Key: YOUR_API_KEY' \
--header 'Content-Type: application/json' \
--data-raw '{
"name": "YOUR_INDEX_NAME",
"dimension": 384,
"metric": "dotproduct",
"pod_type": "s1h",
"index_config": {
"hybrid_search": {
"avg_doc_len": 100
}
}
}'
Note: as of the publishing of this repo, the pinecone-client does not include hybrid API references since it is still in preview status. We have included a temporary class 'hybrid_pinecone_client.py' in this repository to take care of what the pinecone-client would normally do. Once the python-client is updated, we will refactor this repository.
You will next modify the following section of the 'Pinecone_io_generate_embeddings.ipynb' file by adding YOUR_API_KEY, YOUR_EVIRONMENT, and YOUR_INDEX_NAME below.
Pinecone_io_generate_embeddings.ipynb
# initialize an instance of HybridPinecone class
pinecone = HybridPinecone(
api_key = "YOUR_API_KEY", # app.pinecone.io
environment = "YOUR_ENVIRONMENT"
)
# choose a name for your index
index_name = "YOUR_INDEX_NAME"
You can now run all of the cells in the Jupyter notebook. If all is successful, you should see query results at the bottom of the notebook. If you have any issues, please see the troubleshooting section.
Writing and deploying Flask applications is outside the scope of this demo however, here is a quick tutorial to help you get started if your are not familiar with the Flask framework.
The Flask application included in this repository is controlled by the 'app.py' file which configures the web services and prepares the server to perform the search functionality. Templates included in the ./templates directory provide the views of the pages and render results.
Before we run Flask, we must configure the 'app.py' file. Create a copy of app_example.py and rename it to app.py. This file name (app.py) is required by Flask.
Replace YOUR_API_KEY, YOUR_ENVIRONMENT, YOUR_INDEX_NAME, and YOUR_SECRET_KEY with the values of your application. YOUR_SECRET_KEY can be any text value you like.
app.py
# Configure and connect to Pinecone index api_key = "YOUR_API_KEY" pinecone_env = "YOUR_ENVIRONMENT" index_name = "YOUR_INDEX_NAME" pinecone = hybrid_pinecone_client.HybridPinecone(api_key, pinecone_env) pinecone.connect_index(index_name) # Flask settings app = Flask(__name__) app.config['SECRET_KEY'] = 'YOUR_SECRET_KEY'
In order to run this application locally, you must start flask from a terminal in the root directory of this project. Here is one example of activating the current python environment using conda, setting debug properties, and starting Flask:
conda activate pinecone&&export FLASK_ENV=development&&export FLASK_DEBUG=1&&flask run
If all is working correctly, you should be able to visit http://127.0.0.1:5000 and see the search UI. Try some searches while adjusting the alpha value to any number between 0 and 1.
A lower value closer to 0 is considered more lexical or keyword oriented. A higher value closer to 1 is more semantic and gives more weight to the meaning of the words rather than the more exact matching of the terms that lexical is suited for.
Sign up for a free Pinecone Account
Hybrid search early access documentation
Introducing the hybrid index to enable keyword-aware semantic search
Getting Started with Hybrid Search
Pinecone's New Hybrid Search - the future of search?
overall_score = alpha * dense_score + (1-alpha)*sparse_score
sparse_score = (overall_score - alpha * dense_score) / (1-alpha)
Python Flask
Jupyter Notebooks
Some of the most common errors:
1.) Unmet dependencies in your local python environment. These can typically be resolved using PIP and installing the missing module/package.
2.) API key, environment, index name not correct. Compare your notebook to the settings in the Pinecone Console
Although Flask is outside the scope of this repository, you may find help here