Map DocChunks from HierarchicalChunker to section content

[eirikfk]

Thanks for a great framework!
I just started exploring Docling and tested the following snippet, which returns a list of DocChunks, correctly divided into sections with accurate numbering:

from docling.document_converter import DocumentConverter
from docling_core.transforms.chunker import HierarchicalChunker

source = "example.docx"
conv_res = DocumentConverter().convert(source)
doc = conv_res.document

chunks = list(HierarchicalChunker().chunk(doc))

My next goal is to enrich each chunk by e.g. extracting metadata, transforming tables, and adding embeddings etc., before constructing a custom object which is later indexed. I want to do this without LlamaIndex or Langchain.

The DocChunks object just contains title and metadata with references. Are there any methods that help me map the titles to the corresponding content within each section, or would I have to implement the mapping myself?

[vagenas]

The easiest way to map to the section is to just use the headings metadata field, which is a list of the applicable (title and) headings for the given chunk.

BTW: the headings may also well be useful context to pass to your embedding / generation model.

[IoannisMaras]

But there can be chunks that do not have headings . I am also looking for a good way to use export_to_markdown() in a chunk

[eirikfk]

The .docx I'm parsing might be challenging, and I see these issues:

• All DocChunks are of type Paragraph.
• headings field is always None.
• Captions field is always None.
• children list is empty.

Looking at the DocChunk list:

• The first elements are correctly identified as sections (sections, subsections, subsubsections) including numeration, but the headings field isn't populated, and the actual heading is populated in the text-field. The parent for these is #/body.
• The next DocChunks are the text components for each section, with the parent correctly set as the previous text within the section, but they are not connected to the actual title.

When I directly parse the document from docx to markdown, it is able to correctly identify the titles.

The document is sensitive and can’t be shared, but I can create a scrubbed version for debugging if needed?

[vagenas]

@IoannisMaras

But there can be chunks that do not have headings

Indeed. As far as sections are concerned though, with the current implementation, this is only visible via the headings.

I am also looking for a good way to use export_to_markdown() in a chunk

I assume there should be some good generic open-source Markdown parsers, but particularly if you are already using a framework, you can look at MarkdownNodeParser (LlamaIndex) or MarkdownHeaderTextSplitter (LangChain).

[vagenas]

@eirikfk

The document is sensitive and can’t be shared, but I can create a scrubbed version for debugging if needed?

If it is easy for you to provide an anonymized example doc that illustrates the situation, it could help indeed.

[vagenas]

@eirikfk @IoannisMaras also be aware that we are currently working towards some further native chunking capabilities, e.g. taking into account the number of tokens — see #191.