GPU and CPU perfomance seems to be the same

[hufffs]

I conducted tests running several PDFs on various machines (MacBook, WSL 2, Ubuntu) with and without GPU.
The results using only the CPU averaged around 11 seconds, while using the GPU resulted in an average of 10 seconds.
The difference seems very small, even when tested with an RTX 6000.
Is this expected behavior, or am I possibly missing some optimization or configuration?

Thanks

[cau-git]

@hufffs Which version of Docling did you use? We have enabled full accelerator support since version 2.12.0

[davidress-ILW]

@huffs Try a comparison with a larger document. I used a 120 page document for comparison by extracting tables: with CPU it took 621 seconds and with GPU it took 95 seconds. A larger document will likely show a better performance margine and ensure the CUDA is indeed active.

[hufffs]

@cau-git

I was using version 2.10, but I have also tested version 2.12.
What’s more curious is that on the MacBook, the performance is very similar between GPU and CPU, and it’s still better than on other machines where the tests were conducted. Here are the details of the configurations

• WSL 2 (RTX 3060, Core i7 11800H, 64GB RAM)
  CPU = 4s
  CUDA = 1.8s

• MACBOOK M4 (16GB RAM)
  CPU = 0.87s
  MPS = 0.9s

• Hyperstack (1 GPU L40, 28 CPUs, 58GB RAM)
  CPU = 2.4s
  CUDA = 1.2s

[cau-git]

@hufffs are your results normalized by page, or is this on a full document? To see better impact of GPU acceleration I recommend to either feed a long document or multiple documents at once, since there is some initialization cost when you run docling, which will bias the result with very short input.

[hufffs]

@cau-git
I used a simple, identical one-page document for all the tests. What surprises me is that the L40 and RTX 6000 performs worse than the MacBook for the same document