Again, using FaissDocumentStore, if the parameter faiss_index_factory_str is set to "IVF96,Flat", we need to do additional train_index, what is the whole process?

[1716649290]

Again, using FaissDocumentStore, if the parameter faiss_index_factory_str is set to "IVF96,Flat", we need to do additional train_index, what is the whole process?

[anakin87]

(I converted the issue in a discussion, seems more appropriate)

Can you better explain your question?
Are you encountering problems?

[1716649290]

My task goal is to store millions of data using Faiss vector database and accelerate the search through IVFx. By consulting the documentation, I found that when reusing the IVFx type, I need to execute the train_index function first. The problem now is that I am not sure where to execute train_index.

[anakin87]

I'm not an advanced FAISS user, but looking at our API reference, I would say you need to train the index only at the beginning.
Then, for each subsequent write operation, you don't need to train the index.

I found more information on this topic on FAISS wiki.

[1716649290]

Can you give a complete demo using train_index?

[1716649290]

My code is as follows, and I get an error when calling run:（Exception: Exception while running node 'Retriever': Error in faiss::DirectMap::idx_t faiss::DirectMap::get(faiss::DirectMap::idx_t) const at /project/faiss/faiss/invlists/DirectMap.cpp:81: direct map not initialized Enable debug logging to see the data that was passed when the pipeline failed.）

def handle_documents(documents, knowledge_path, query_embedding_model,passage_embedding_model):
    knowledge_store = FAISSDocumentStore(sql_url=f"sqlite:///{knowledge_path}",
                                         faiss_index_factory_str="IVF96,Flat",
                                         batch_size=10000,
                                         return_embedding=True)
    retriever = DensePassageRetriever(
        document_store=knowledge_store,
        query_embedding_model=query_embedding_model,
        passage_embedding_model=passage_embedding_model,
        use_gpu=True,
        embed_title=True,
        batch_size=256, max_seq_len_query=128, max_seq_len_passage=512,
        devices=["cuda:0"]
    )
    if documents is not None and documents != "":
        dataset = []
        for doc in documents:
            dataset.append(
                Document(
                    content=doc["content"],
                    meta={"title": doc["title"]}
                )
            )
        # The metadata is preprocessed
        processor = PreProcessor(
            clean_empty_lines=True,
            clean_whitespace=True,
            clean_header_footer=True,
            split_by="word",
            split_length=648,
            split_overlap=32,
            split_respect_sentence_boundary=True
        )
        documents = processor.process(dataset)
        a = retriever.embed_documents(documents)
        pipeline = Pipeline()
        knowledge_store.train_index(embeddings=a)
        knowledge_store.write_documents(documents, batch_size=256, duplicate_documents="overwrite")
        knowledge_store.update_embeddings(retriever=retriever, batch_size=256, update_existing_embeddings=False)
        pipeline.add_node(component=retriever, name='Retriever', inputs=['Query'])

        results = pipeline.run(
            "What is Chronic Kidney disease",
            params={"Retriever": {"top_k": 20}}
        )['documents']

[anakin87]

Ok, I managed to reproduce your issue.

I think it can be fixed with
knowledge_store.faiss_indexes['document'].make_direct_map()

before querying.

LMK if it helps.

[1716649290]

Thank you very much for your reply. After I made the following modifications to the code, I am currently encountering the following problems, Through debugging, I found that the number in final_documents is 5142, but the number in documents_indexed is 5136. How is this caused?:

“ValueError: The number of documents in the SQL database (5136) doesn't match the number of embeddings in FAISS (5142). Make sure your FAISS configuration file points to the same database that you used when you saved the original index.”

        embed_documents = retriever.embed_documents(pro_documents)
        middle_documents = []
        for i, doc in enumerate(pro_documents):
            middle_documents.append(
                Document(
                    content=doc.content,
                    meta={"title": doc.meta['title']},
                    embedding=embed_documents[i]
                )
            )
        final_documents = documents_with_embeddings(middle_documents)
        knowledge_store.train_index(final_documents)
        knowledge_store.faiss_indexes[knowledge_store.index].make_direct_map()
        for i in range(0, len(final_documents), 10000):
            knowledge_store.write_documents(final_documents[i: i + 10000] ,duplicate_documents="overwrite")
        documents_indexed = knowledge_store.get_all_documents()
        print(f"documents_indexed:{len(documents_indexed)}")

def documents_with_embeddings(documents):
    # drop documents without embeddings from the original fixture
    return [d for d in documents if d.embedding is not None]

[anakin87]

This seems like a different problem. I would suggest recreating your index from scratch and making your flow work with a small number of Documents, first.