InMemoryDocumentStore does not support filters for BM25 retrieval. This parameter is ignored.

[LeoTungAnh]

Dear all,

I am trying to run all tutorials with InMemoryDocumentStore. Although existing documents said that this kind of storing document works with BM25, but actual running code does create an output of unsupporting BM25 for InMemory

[anakin87]

Hey @LeoTungAnh!

• Are you using Haystack 1.x or 2.x?
• Can you provide a code example, showing the obtained result and the expected one?

[LeoTungAnh]

Hi @anakin87 ,
I am using haystack 1.x by installing farm-haystack via pip.

The code is following:

Get document store and retriever

from haystack.document_stores import InMemoryDocumentStore
from haystack.nodes import BM25Retriever

document_store = InMemoryDocumentStore(use_bm25=True)
retriever = BM25Retriever(document_store=document_store)

Do some logic to add document to document store here

Perform query

item_id = "B0074BW614"
query = "Is it good for reading?"
retrieved_docs = retriever.retrieve(query=query, top_k=3, filters={"item_id":[item_id], "split":["train"]})

When I call retriever, its outputs is following:
InMemoryDocumentStore does not support filters for BM25 retrieval. This parameter is ignored.

I also create a pipeline to test:
from haystack.nodes import FARMReader
from haystack.pipelines import ExtractiveQAPipeline

model_ckpt = "deepset/minilm-uncased-squad2"
reader = FARMReader(model_name_or_path=model_ckpt, use_gpu=True, return_no_answer=True)
pipe = ExtractiveQAPipeline(reader, retriever)
labels_agg = document_store.get_all_labels_aggregated(index="label", open_domain=True, aggregate_by_meta=["item_id"])
metrics = eval_results.calculate_metrics()
print(f"Recall for Retriver: {metrics['Retriever']['recall_single_hit']:.2f}")
--> output: 0.58

When I use the Dense Retriever, the f1 score is around 0.87

[anakin87]

OK. You are right. Filters don't work with BM25Retriever + InMemoryDocumentStore in Haystack 1.x.

If you are approaching Haystack for the first time, I would suggest starting with Haystack 2.0-beta. We will release the stable version in a few weeks.

Anyway, if you explain what you are trying to do, I can provide some suggestions...

[LeoTungAnh]

Thanks for your reply!
I have made decision to use Dense Retriever. However, I want to test performance of different approaches for Retriever. For this point, using InMemory is the fastest way for me, which help me to reduce complexity of installing a docker or real database.