How do I delete all documents from a Store (chroma, in this case) before adding new ones using pipeline method

[rmrbytes]

Hello:

I have just discovered this brilliant library and struggling on one aspect viz. removal of docs/embeddings from a store on an update.

If I have set the store as below

    # set the store
    document_store = ChromaDocumentStore(collection_name=f"agent_{agent_name}", persist_path=store_path, distance_function=settings.distance_metric)
    document_writer = DocumentWriter(document_store=document_store)

and later use the pipeline to add new ones as below:

# add components to the pipeline
    preprocessing_pipeline = Pipeline()
    # router
    preprocessing_pipeline.add_component(instance=file_type_router, name="file_type_router")
    # document converters
    preprocessing_pipeline.add_component(instance=text_file_converter, name="text_file_converter")
    preprocessing_pipeline.add_component(instance=markdown_converter, name="markdown_converter")
    preprocessing_pipeline.add_component(instance=pdf_converter, name="pypdf_converter")
    # joiner
    preprocessing_pipeline.add_component(instance=document_joiner, name="document_joiner")
    # cleaner
    preprocessing_pipeline.add_component(instance=document_cleaner, name="document_cleaner")
    # splitter
    preprocessing_pipeline.add_component(instance=document_splitter, name="document_splitter")
    # embedder
    preprocessing_pipeline.add_component(instance=document_embedder, name="document_embedder")
    # document writer
    preprocessing_pipeline.add_component(instance=document_writer, name="document_writer")

    # connect the router with sources
    preprocessing_pipeline.connect("file_type_router.text/plain", "text_file_converter.sources")
    preprocessing_pipeline.connect("file_type_router.application/pdf", "pypdf_converter.sources")
    preprocessing_pipeline.connect("file_type_router.text/markdown", "markdown_converter.sources")

    # connect components with document joiner
    preprocessing_pipeline.connect("text_file_converter", "document_joiner")
    preprocessing_pipeline.connect("pypdf_converter", "document_joiner")
    preprocessing_pipeline.connect("markdown_converter", "document_joiner")

    # connect the document joiner with cleaner
    preprocessing_pipeline.connect("document_joiner", "document_cleaner")
    # connect cleaner with splitter
    preprocessing_pipeline.connect("document_cleaner", "document_splitter")
    # connect splitter with embedder
    preprocessing_pipeline.connect("document_splitter", "document_embedder")
    preprocessing_pipeline.connect("document_embedder", "document_writer")

    # run the pipeline
    result = preprocessing_pipeline.run({"file_type_router": {"sources": list(Path(dir).glob("**/*"))}})

In my use-case, the directory may have some new files added and some removed. Hence I would want those embeddings/chunks removed from the store.

Thanks

[davidsbatista]

You can delete all the documents before starting the pipeline: doc_store.delete_documents() - that would be the easiest solution.

[rmrbytes]

Thanks @davidsbatista . I guess I was misled by the filter list in the documentation

def delete_documents(document_ids: List[str]) -> None

[kirahman2]

@rmr-code Did the solution work for you?

@davidsbatista I'm using elasticsearch. When I attempt to run document_store.delete_documents() in my code I get the error

TypeError: ElasticsearchDocumentStore.delete_documents() missing 1 required positional argument: 'document_ids'

When I attempt to run it according to the error's instructions

document_store.delete_documents('e75c955a2a742479f9c4b97f5b5c32bfa8026b0644e11e62097045bb3062cf08')

or as

document_store.delete_documents(document_ids='e75c955a2a742479f9c4b97f5b5c32bfa8026b0644e11e62097045bb3062cf08')

The document doesn't actually get deleted.

This is my full code

document_store.delete_documents('e75c955a2a742479f9c4b97f5b5c32bfa8026b0644e11e62097045bb3062cf08')

list_temp = document_store.filter_documents()
for val in list_temp:
    print(val)

output:
Document(id=e75c955a2a742479f9c4b97f5b5c32bfa8026b0644e11e62097045bb3062cf08, content: '2 (cut off on purpose)

Is this a potential bug? Or am I using this incorrectly?

[kirahman2]

I reviewed the source code and found the correct syntax.

document_store.delete_documents(document_ids=['insert doc id here>'])

[rmrbytes]

Something like this works

    ...
    document_store = ChromaDocumentStore(collection_name="something")
    ....
    # print("before:", document_store.count_documents())
    # get all docs with blank filter
    docs = document_store.filter_documents()
    # extract ids
    ids = [d.id for d in docs]
    # send them to delete method
    document_store.delete_documents(ids)
    # print("after", document_store.count_documents()) # shows 0

In my test I had only 1 collection. will check out with multiple collections in the store later and revert in case of any issues.

[davidsbatista]

sorry, I over looked the docs you need to specifiy the documents you want to delete, and you can get them all with .filter_documents() as @rmr-code shows above

[bastianhaarmann]

For me document_store.delete_documents() does not work with ElasticSearch, possibly because my documents are split by the NLTKDocumentSplitter and all chunks have the same doc.id

print("before:", document_store.count_documents())
# get all docs with blank filter
docs = document_store.filter_documents()
# extract ids
ids = [d.id for d in docs]
# send them to delete method
document_store.delete_documents(ids)
print("after", document_store.count_documents())

before: 3
after 3

[davidsbatista]

The NLTKDocumentSplitter is now deprecated in favor of the DocumentSplitter, which can now achieve the same functionality as the other, i.e.: split_by=sentence.

Does the same issue occur if you use the DocumentSplitter?