Using ruby_llm for PDF Question-Answering with Precise Source Locations

[elalemanyo]

Hi everyone,

I’m exploring the idea of using ruby_llm to enable question-answering on a set of PDFs while also providing precise source locations (e.g., “this answer came from page 7, paragraph 2”).

I reached out to @crmne, who shared some great insights:

• Direct PDF analysis: RubyLLM supports using Claude and Gemini models for PDF-based Q&A, but their ability to return accurate source locations is limited.
• Chunk-based retrieval: A more reliable approach might involve:
  • Splitting PDFs into smaller sections while preserving metadata (page number, paragraph location).
  • Storing these sections in a vector database (e.g., pgvector).
  • Using an LLM to answer questions based on retrieved chunks while referencing the source.

I’m curious — has anyone here built something like this before using ruby_llm or a similar setup?
What tools or libraries did you find most useful for PDF chunking and embedding in a Rails-based project?
Are there any recommended strategies to ensure accurate source attribution in the responses?

Would love to hear about any experiences, challenges, or best practices! 🚀

[earnestaddae]

@elalemanyo for developing a RAG system, you can use the following gems in my experience.

• baran for text splitting and chunking
• pdf-reader for text extraction from .pdf files
• neighbor for nearest neighbor search of your embeddings
• ruby_llm can handle the embeddings. The documentation is straightforward

I don't know which database you are using or intend to use, but I will recommend pgvector for storing your embeddings if you using postrgres

PS: I'm yet to use ruby_llm gem, but I don't think it's going to be any different from using open_router or any of the several ruby llm integration.

I hope this is helpful

[crmne]

There's an issue about support for Anthropic Citations, hope that helps! #52