Model Performance Limited by Single Core CPU Speeds - GPU Not Fully Utilized

[Offek]

Hello!

I've been playing around with spaCy and have created multiple SpanCat, TextCat, and NER models using curated-transformers that run on a GPU.

When either training or running the models, the performance is dictated by the CPU single-core performance.
There is always a single CPU core running at 100% and the GPU utilization hits 40% on average.

I've tested this on multiple machines with different CPUs and the same GPU (RTX 3090), and it proves to be true; the weaker the single-core performance of the CPU, the lower the GPU utilization and thus the slower the performance.

How could this be fixed? Multithreading is currently not a good solution for GPU since the VRAM is scarce. I've gotten a stronger CPU just so the training and execution of the models would be faster, but I believe there is some kind of bottleneck. Solving it could easily result in at least a 2X improvement.

I've tried modifying hyperparameters such as batch_size with no luck.

Please let me know if this is intended, or perhaps there is a known workaround.
Thank you!

[lukas-vollmer]

We solved it by creating threads around the spacy training (e.g with import multiprocessing) and train multiple folds simultaneously (run on gpu).