Add S3 integration for input/output files

[willkara]

Requested feature

A lot of groups make heavy use of S3 compatible endpoints like S3 itself and LakeFS to store their documents and artifacts. I was hoping that the team would be able to take a look at implementing it as a possible integration for storage.

Thought is that teams could pass an S3 compatible bucket as a source/destination and docling would automatically read in user credentials and configuration for reading/writing to the bucket for results.
...

Alternatives

None
...

[dolfim-ibm]

You might be interested in trying out Data-Prep-Kit which will orchestrate a distributed batch conversion using Docling, and it has already support for s3 storage: https://ds4sd.github.io/docling/integrations/data_prep_kit/

Connectors directly in Docling might also be supported in the future, but there you can get something already working.

[willkara]

O snap, that's awesome, will definitely check it out. Running manual test-cases and 5000+ file conversions manually for now in DevSpaces, so I can prove it out for wider usage.

Data-Prep-Kit seems like it's more of an orchestrator for these kinds of workloads using the individual tools. Given that, I wonder if Docling would/should focus on agnostic integrations, rather than direct support. Thanks!