Issues of loading the full pdf document in cloud qdrant

[marcello-calabrese]

Hi, I am new with qdrant, so maybe what I am saying is not 100% correct :-).

I am using the free tier of qdrant cloud, I am trying to upload a pdf document in the vector db, but for what I can see it only load one chunk. Not sure if it is a problem on how I configure the size or the points with Point struct, but see below the code I use and the only Point with Id 0 I can see on Qdrant cloud. Thanks a lot, Marcello

Code:

# Create a connection to Qdrant

qdrant_client = QdrantClient(
    url = os.getenv("QDRANT_HOST"),
    api_key=os.getenv("QDRANT_API_KEY"),
    )

# now we create the OpenAI object

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

MODEL = "text-embedding-3-large" # we add the OpeanAI embedding model 

# create a list to store the points

for chunk in chunks:
    # create the embeddings for each chunk
    embeddings = client.embeddings.create(model=MODEL, input=chunk)

    # create the point struct with the embeddings and the payload and upload it to the collection
    points = [
        PointStruct(
            id=idx,
            vector = data.embedding,
            payload = {"text": chunk},
            )
        for idx, (data, chunk) in enumerate(zip(embeddings.data, chunks))]

# Create a collection

COLLECTION_NAME = "rag_mageai"

qdrant_client.recreate_collection(
    collection_name=COLLECTION_NAME,
    vectors_config=models.VectorParams(size=3072, distance=models.Distance.COSINE),
    )

print('Create collection response:', qdrant_client)

# get information about the collection

collection_info = qdrant_client.get_collection(collection_name=COLLECTION_NAME)

print('Collection info:', collection_info)

'''4. Insert them into the collection'''


operation_info = qdrant_client.upsert(
    collection_name=COLLECTION_NAME,
    wait=True,
    points=points
)           

print('Upsert operation info:', operation_info)

Only One chunk on Qdrant cloud:

[Mohana-Murugan]

Hi there,

If it is PDF file:
Split the PDF into individual pages, convert each page to text, and treat each page as a separate document. Store each document in your vector database, along with any necessary metadata (e.g., page number, document ID) to track which page belongs to which document.