Production deployment via HuggingFace

[kdaniel21]

Hi, first of all, thank you for the project, I've been testing it in the last few days, and it seems perform great!

We'd love to try Docling in production, but we were wondering about what the best way would be to deploy it. It doesn't need to "hyperscale" to hundreds of documents per minute, but we still want to make sure that it can deal with a dozen of documents per minute, and it doesn't crash for smaller spikes.

The "intuitive" idea i had was deploying it on a "beefy" VM with dedicated GPUs, and running it from there as usual (probably with a queue to avoid overloading it). However, I was wondering if it'd possible to deploy the models (EasyOCR, layout recognition, table extraction etc.) separately e.g. on SageMaker/AzureML through their HuggingFace integration, and run the rest of the pipeline run on a "regular" CPU-only VM. This would allow scaling the computationally heavy tasks independently from the code that's responsible for executing the pipeline, and thus could be easily integrated into an existing service/backend as one would only have to install the docling package, deploy the models, and use the "managed models" for inference.

Additionally, this would also make it more accessible to use it locally, as one could just deploy the pre-trained models remotely, and use them even with tighter hardware constraints (despite docling having a great performance even on consumer grade hardware!).

I also saw that there was a docling-ibm-models repo, which seems to contain the actual code for the models, and also docling-serve, which seems to be a simple API wrapper for docling.

Thanks for taking the time, and looking forward to using Docling in production!

[cau-git]

Hi @kdaniel21, great to see you thinking towards production usage of docling! I am happy to discuss further with you on how docling can be leveraged in production scenarios. As a starting point, let me share with you a few pieces that may give you relevant context.

1. Some background: We have distilled the docling package out of Deep Search, a sophisticated platform for document knowledge extraction, which we have been engineering over the past several years in IBM Research. It is a research asset which can be licensed commercially, and has built-in all the distributed scaling capabilities, can deploy models on large-scale cloud infrastucture, and more. It is the result of years of systems and AI engineering research, which we have also published about here: https://arxiv.org/abs/2206.00785.
   One important realization from this work is that the I/O and network overhead of leveraging distributed/remote models can be very significant, and a more monolithic solution with all models running locally where the input documents first arrive, will in many scenarios beat a highly distributed processing scheme. This is at least valid if you have highly optimized and computationally moderately expensive models, such as we provide in docling so far (OCR is the exception here).

2. You can get a good idea of how to run docling at massive scale by checking out IBM's data-prep-kit. data-prep-kit is an open-source toolkit to scale data pre-processing (e.g. for LLM training) with so-called transforms, such as this one using docling. It supports running either locally or on a ray cluster, where each ray worker will own an independent instance of a docling converter. We are successfully using it to bulk-process PDF libraries across hundreds of nodes.

[kdaniel21]

Hi @cau-git, thank you for elaborate response!
First of all, thanks for the paper, I'll give it a read. Your research seems to be indeed pretty interesting and relevant. :)

To be a bit more specific with our use case, we have two priorities:

1. We want to integrate Docling into our architecture/infrastructure as simply as possible. We have a monolith in a language other than Python, and therefore the easiest would be:
   1. Creating a "binding" for the docling (Python) library to do everything that's "computationally cheap" as part of our monolith (likely in the same process).
   2. Deploying the models on a serverless inference platform (SageMaker, HF, AzureML), so that the Docling binding from point 1 can call these.
2. Performance to the extent where the service runs reliably and doesn't crash even if a longer document comes in. Having a few seconds of network overhead doesn't matter, and we could partially solve this with a e.g. a queue.

I can see how a monolithic approach leads to better performance in this case, but we fear that in our case, this comes at the cost of infrastructure complexity:

• Either we'd run the whole thing in the same process as our monolith, and have to allocate more resources, and risk e.g. crashes*.
• We'd have to deploy a service resembling docling-serve that can scale independently from our monolith (with more resources). This would work, just requires us to maintain/monitor/manage one more service.

*: My performance concerns also come from my local experience. I'm using normal M2, and with ~8 short (1-5 pages) documents, the process constantly crashed. This could be mitigated with queuing, but our production instances are running with even fewer resources (AWS ECS, scale vertically if needed). This makes me think that it could cause problems there (especially with peaks), and having the option to simply use hosted models would make local development also easier.

While this is specific to us, I believe that this is actually a common use case where people want to add document extraction to their existing system as easily as possible. The scale is not big enough to justify a completely separate service, but at the same time running it in the "main" service isn't a realistic option either, especially with OCR (e.g. for a "regular REST API"). Then, using a managed service is probably the cheapest option (almost no maintenance, high reliability).

Thanks again for taking the time! I wanted to give some more context to our use case and why we believe that deploying it on a serverless platform could be a common use case.
Hence the question is:

• do you have any recommendations to integrate Docling that's reliable and simple?

As I said in the other GH issue, we'd be ready to try to contribute depending on the scope.

[PeterStaar-IBM]

@kdaniel21 To follow uo with what @cau-git said, we are currently writing a 2nd version of the technical report, which will include a lot more benchmarking on a variety of hardware. Could you inform us where you intend to run it, specifically the specs (pure cpu, GPU accelerated, etc). This could give us a good indication on what to benchmark.

[LuoTengBin]

Hi @pavel-denisov-fraunhofer, thank you very much for your work! Inspired by you, I tried to convert the tableformer model to pytorch format, which can work properly, but further converted it to openvino format failed, I wonder if you have any good ideas? Or have you tried it before?

[pavel-denisov-fraunhofer]

Hi @LuoTengBin, thanks for your interest! I have not tried to convert the tableformer model. If conversion from PyTorch to OpenVINO fails, you might try to convert from PyTorch to ONNX first, and then from ONNX to OpenVINO.

[LuoTengBin]

Hi @pavel-denisov-fraunhofer, thanks for your reply! According to the ideas you provided, I tried to export pytorch to onnx first, but unfortunately, it also failed. The failure information is as follows:

Here is the code I used to convert the model format:
`import torch
import openvino as ov

core = ov.Core()
model = torch.load("D:\AIGC\docling-main\ds4sd\model_artifacts\tableformer\table_model.pt")
torch.onnx.export(model, (torch.rand(1, 3, 224, 224), ), 'table_model.onnx')
ov_model = ov.convert_model('table_model.onnx')
ov.save_model(ov_model, "table_model.xml")`

[pavel-denisov-fraunhofer]

I see. I'm not familiar with the architecture of Tableformer. It appears that it has 2 separate models inside: ResNet image encoder (model._encoder) and Transformer Encoder-Decoder (model._tag_transformer). The Transformer itself contains 2 big models which are used in the prediction separately: Encoder and Decoder (model._tag_transformer._encoder and model._tag_transformer._decoder). There are also smaller parts of the Transformer, but probably it's not worth optimization at the first try.

You can try to convert model._encoder, model._tag_transformer._encoder, and model._tag_transformer._decoder to OpenVINO. The intermediate ONNX conversion is probably not needed in this case.

[LuoTengBin]

Ok, thank you very much for your advice, I will try it.