Key value pairs? Forms?

[ericfeunekes]

I'm curious if you have thoughts on how to use docling for things like completed government forms? E.g extract all key value pairs from a page.

It seems like a difficult problem because a table recognition model could easily get confused between tables and KV pairs. But I'm bringing it up because the extensibility of this library seems like it offers a great opportunity to build something like this that actually works, particularly with the JSON output.

I'm not aware of any specific model that can do it just yet, but even something like a moderately powerful VLM could be inserted in the pipeline somewhere to predict the KV pairs elements.

So have you thought about how to integrate this? How would you build it into the pipeline somewhere as a prediction, even using a powerful closed source model as a proof of concept initially?

[VickySekhon]

This is a great idea, I think that Docling could definitely tackle this issue by creating a specific pipeline targeting government forms as you mentioned. The technology already exists, I believe it's simply a matter of tailoring the output.

[ericfeunekes]

Yeah. It's not even necessarily government forms, but forms generally. So more like key value pair extraction.

I think the important part is differentiating between tables and KV pairs, and then figuring out what the output format would be.

Should this be an issue? It'd be a major help for some stuff I'm working on so happy to test some things out.

[PeterStaar-IBM]

@VickySekhon @ericfeunekes Yes, we are actively working on a form model for exactly this reason. Please stay tuned!

If you want to collaborate with us (eg share some of the forms you are interested in ), please let us know. We would be happy to do so.

[ericfeunekes]

I have thousands of different forms that I'm looking to run this on. How do you want to connect.

[PeterStaar-IBM]

contact us here: [email protected]

[ericfeunekes]

Here are some forms you can try from the Canadian tax authority. You can get French and English by clicking the language in the top right

https://www.canada.ca/en/revenue-agency/services/forms-publications/forms.html

[PeterStaar-IBM]

sweet! working on a quick PR (DS4SD/docling-parse#58) to expose the annotations/form-fields directly from the PDF (when they have them).

[ericfeunekes]

Here are UK forms:
https://www.gov.uk/self-assessment-tax-return-forms
https://www.gov.uk/government/collections/self-assessment-helpsheets-main-self-assessment-tax-return

[ericfeunekes]

US: https://www.irs.gov/forms-instructions-and-publications

Those US forms list alternative languages directly and it looks like there are around a dozen different languages available. This could be a really great cross lingual dataset to validate that questions in one language can extract data from forms in other languages. There are several unstructured text documents there too, which may help with things like cross lingual chunking, etc.

[ericfeunekes]

@PeterStaar-IBM any progress you can share on this? I saw the recent PR you shared but have been occasionally testing form extraction and haven't seen a change.

[PeterStaar-IBM]

@ericfeunekes Actually, I am downloading all the forms you listed above and trying to do some regression tests now. I have also my team-members working on propagating the key-value and form regions.

So, not forgotten at all and working behind the scenes to get a good first version!