Issue: RAG with LangFlow & AstraDB - Multi-PDF Querying Leads to Hallucinations

[ealss101]

Problem Statement

I've been implementing Retrieval-Augmented Generation (RAG) using LangFlow and AstraDB. While most tutorials focus on handling a single PDF, I need to upload multiple PDFs and query them individually. However, I'm facing an issue where the LLM hallucinates—when I ask about a specific document, it sometimes pulls irrelevant information from other PDFs.

What I’ve Tried

• Successfully ingesting multiple PDFs into AstraDB and connecting it with LangFlow.
• Querying the database using vector search to retrieve the most relevant document chunks.
• Implementing similarity search for better retrieval.

Issue Observed

• The LLM sometimes mixes content from multiple PDFs instead of keeping responses isolated to the specific document I’m asking about.
• Even if I query about "Document A," the agent occasionally pulls facts from "Document B" or "Document C."

Potential Solution?

From discussions on Reddit, I've seen recommendations to update the metadata column in AstraDB. Specifically:

• Adding a "metadata" field in astraDB collections to associate chunks with their original document name.
• Modifying retrieval logic to filter results by filename or document ID before passing them to the LLM.

Help Needed

1. Best Practices for Metadata Usage – How should I structure metadata in AstraDB to ensure better filtering?
2. LangFlow Configuration – Is there a way to adjust LangFlow’s pipeline to prioritize correct document retrieval?
3. Chunking Strategy – Could the hallucinations be a result of poor chunking? Should I adjust chunk sizes?

Would appreciate any insights or example implementations from the community! 🙌

[dosubot[bot]]

To address the issues you're facing with Retrieval-Augmented Generation (RAG) using LangFlow and AstraDB, here are some suggestions:

1. Best Practices for Metadata Usage:

   • Metadata Structuring: Ensure that each document chunk in AstraDB has metadata fields that uniquely identify the document it belongs to, such as document_name or document_id. This will help in filtering the results accurately during retrieval.
   • Metadata Indexing: Use the metadata_indexing_include option in LangFlow to specify which metadata fields should be indexed. This can improve the efficiency of filtering based on metadata ^[1].

2. LangFlow Configuration:

   • Filtering by Metadata: You can adjust LangFlow's retrieval logic to filter results by metadata before passing them to the LLM. Modify the SelfQueryRetrieverComponent to include filtering logic based on metadata fields like document_name or document_id. This ensures that only relevant document chunks are considered ^[2].
   • Advanced Search Filter: Utilize the advanced_search_filter in LangFlow to refine your search queries based on metadata. This can help in isolating responses to specific documents ^[3].

3. Chunking Strategy:

   • Adjusting Chunk Sizes: The default chunking strategy in LangFlow uses a chunk_size of 1000 and a chunk_overlap of 200. If hallucinations persist, consider experimenting with different chunk sizes and overlaps to see if it improves the accuracy of document retrieval ^[4].

By implementing these strategies, you should be able to reduce hallucinations and improve the accuracy of document-specific responses in your RAG setup.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}