How to pass documents from retrieval to DynamicChatPromptBuilder?

[denyslazarenko]

I am using Haystack v2.0 and I am trying to design the RAG with chat functionality. I came up with the pipeline below.
The step which is not working currently is pass documents from retrieval to prompt_builder. I am trying something like this query_pipeline.connect("retriever", "prompt_builder.template_variables.documents") but I am getting the following error PipelineConnectError: 'prompt_builder.template_variables.documents does not exist. Input connections of prompt_builder are: prompt_source (type List[ChatMessage]), template_variables (type Optional[Dict[str, Any]]).

I have feeling that this is not a proper usage, but I could not find in documentation clear explanation on how to do it, could you help me with this?
One possible solution which I see is to separate it into two pipeline and then form messages with retrieved documents.

import os
from haystack.components.generators import OpenAIGenerator
from haystack import Pipeline
from haystack.components.builders.answer_builder import AnswerBuilder
from haystack.components.builders.prompt_builder import PromptBuilder
from haystack_integrations.components.retrievers.pgvector import PgvectorEmbeddingRetriever
from haystack.components.embedders import OpenAITextEmbedder
from haystack.dataclasses import ChatMessage
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.components.builders import DynamicChatPromptBuilder


def get_query_pipeline(document_store, k=10):
    text_embedder = OpenAITextEmbedder()
    retriever = PgvectorEmbeddingRetriever(document_store=document_store, top_k=k)
    prompt_node = DynamicChatPromptBuilder()
    # llm = OpenAIGenerator()
    llm = OpenAIChatGenerator(model="gpt-3.5-turbo")
    answer_builder = AnswerBuilder()

    query_pipeline = Pipeline()
    query_pipeline.add_component("text_embedder", text_embedder)
    query_pipeline.add_component("retriever", retriever)
    query_pipeline.add_component("prompt_builder", prompt_node)
    query_pipeline.add_component("llm", llm)
    query_pipeline.add_component("answer_builder", answer_builder)

    query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
    query_pipeline.connect("retriever", "prompt_builder.template_variables.documents")

    query_pipeline.connect("prompt_builder.prompt", "llm.messages")
    query_pipeline.connect("llm.replies", "answer_builder.replies")
    query_pipeline.connect("llm.meta", "answer_builder.meta")
    query_pipeline.connect("retriever", "answer_builder.documents")

    return query_pipeline


def answer_question(query_pipeline, question):
    prompt_template = """
        Given these documents, answer the question.\nDocuments:
        {% for doc in documents %}
            {{ doc.content }}
        {% endfor %}

        \nQuestion: {{query}}
        \nAnswer:
        """
    messages = [ChatMessage.from_system("You are a helpful bot bot to answer questions."),
                ChatMessage.from_user("{{ query }}"),
                ChatMessage.from_system(prompt_template)
                ]

    result = query_pipeline.run({
        "text_embedder": {"text": question},
        # "retriever": {"filters": {"field": "meta.year", "operator": "›", "value": 2015},},
        "prompt_builder": {"template_variables": {"query": question},
                           "prompt_source": messages},
        "answer_builder": {"query": question}
        },
    )
    return result

[anakin87]

Hello, @denyslazarenko!

It seems that you have to declare the runtime_variables (such as Documents) when initializing the DynamicChatPromptBuilder,
as you can read in the docs.

LMK if this helps...

[denyslazarenko]

Hi @anakin87 ,

Thank you very much for your quick reply and yes that helps a lot :)

Maybe one point which would be helpful to provide in the documentation is that only last message will be treated as template:
before i tried to construct messages as this:

messages = [ChatMessage.from_system("You are a helpful bot bot to answer questions."),
            # ChatMessage.from_system("Documents: {{ documents }} "),
            ChatMessage.from_user("Documents: {{ documents }} Query: {{ query }}")
            ]

    def _process_chat_messages(self, prompt_source: List[ChatMessage], template_variables: Dict[str, Any]):
        """
        Processes a list of :class:`ChatMessage` instances to generate a chat prompt.

        It takes the last user message in the list, treats it as a template, and renders it with the provided
        template variables. The resulting message replaces the last user message in the list, forming a complete,
        templated chat prompt.

One additional question which I have: in AnswerBuilder input List[str] is expected, but OpenAIChatGenerator returns list[ChatMessages], the workaround is to create a new class to parse it there, is that correct?

@component
class AnswerBuilderFromChatMessage:

[anakin87]

(FYI @dfokina: a suggestion to improve docs ☝️)

@denyslazarenko I think you are right...

[Filocava99]

Sorry for reopening the issue but the output of the retriever is a list of documents while the input for the runtime variables is a dictionary. How am I supposed to connect them?
query_pipeline.connect("retriever.documents", "prompt_builder.template_variables") obviously gives a type mismatch error

[anakin87]

Hey, can you post an example of your Pipeline?

[Filocava99]

Hi, here it is:

query_pipeline = Pipeline()
query_pipeline.add_component("text_embedder", OpenAITextEmbedder(model=os.environ["EMBEDDINGS_MODEL"], dimensions=1536))
query_pipeline.add_component("retriever", ChromaEmbeddingRetriever(document_store=document_store))
query_pipeline.add_component("prompt_builder", DynamicChatPromptBuilder())
query_pipeline.add_component("llm", OpenAIChatGenerator(model="gpt-3.5-turbo-0125"))  # model="gpt-4-0125-preview"
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
query_pipeline.connect("retriever.documents", "prompt_builder.runtime_variables")
query_pipeline.connect("prompt_builder.prompt", "llm.messages")

[anakin87]

query_pipeline = Pipeline()
query_pipeline.add_component("text_embedder", OpenAITextEmbedder())
query_pipeline.add_component("retriever", ChromaEmbeddingRetriever(document_store=document_store))
query_pipeline.add_component("prompt_builder", DynamicChatPromptBuilder(runtime_variables=["documents"]))
query_pipeline.add_component("llm", OpenAIChatGenerator(model="gpt-3.5-turbo-0125"))
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
query_pipeline.connect("retriever.documents", "prompt_builder.documents")
query_pipeline.connect("prompt_builder.prompt", "llm.messages")

this works, but agree that an example in the docs would help...

[Filocava99]

Thanks! I was missing the runtime variable definition in the constructor, it wasn't much clear in the documentation :/

[julian-risch]

@Filocava99 We've updated the docs and added an example for using DynamicChatPromptBuilder with runtime variables: https://docs.haystack.deepset.ai/v2.0/docs/dynamicchatpromptbuilder#in-a-pipeline-with-runtime-variables