[Docker] Increasing image size

[RobinFrcd]

🐛 Describe the bug

Hi,
The docker image sizes got multiplied by almost 5 over the last few releases:

• 0.6.0-cpu: 496.91 MB
• 0.7.0-cpu: 1.75 GB
• 0.8.0-cpu: 2.37 GB

Biggest folders are:

2.6G    /home/venv/lib/python3.9/site-packages/nvidia
713M    /home/venv/lib/python3.9/site-packages/torch
185M    /home/venv/lib/python3.9/site-packages/triton

425M    /usr/lib/jvm
346M    /usr/lib/x86_64-linux-gnu
97M     /usr/lib/gcc

Are they all needed ? Especially the Nvidia one, as we're running on CPU ?

This is really slowing down scaling in production.
Is it a known issue ?

Thanks,

Error logs

Installation instructions

Docker

Model Packaing

config.properties

No response

Versions

0.8.0-cpu

Repro instructions

Possible Solution

No response

[agunapal]

@RobinFrcd Thanks for reporting. This is something we have been tracking and trying to improve.

One of the challenges is that PyTorch binary sizes have increased with each release. Since we install PyTorch, we see a corresponding increase in TorchServe docker image size.

However, we did switch from dev to prod image in 0.8.0, which helped reduce the size.

We are working on addressing #2358 which will reduce the CPU image size.

We also have a broader effort to make torchserve's mandatory dependencies smaller. We will update the community when we have made progress on this.

[msaroufim]

cc @atalman

[malfet]

It clearly looks like 0.7.0-cpu are not installing cpu PyTorch binaries but a GPU ones.
@msaroufim what's the mapping between torchserve and pytorch releases?
Quick glance at https://github.com/pytorch/serve/blob/master/ts_scripts/install_dependencies.py shows that CUDA-accelerated flavor of torch would be installed inside cpu docker image.

[malfet]

Looks like pytorch-cpu wheel file increase from 1.12 to 2.0.0 is less than 10Mb (i.e. around 3%):

$ pip install torch==1.12.0 --index-url https://download.pytorch.org/whl/cpu/
  Downloading https://download.pytorch.org/whl/cpu/torch-1.12.0%2Bcpu-cp39-cp39-linux_x86_64.whl (189.0 MB)
$ pip install torch==1.13.0 --index-url https://download.pytorch.org/whl/cpu/
  Downloading https://download.pytorch.org/whl/cpu/torch-1.13.0%2Bcpu-cp39-cp39-linux_x86_64.whl (198.5 MB)
$ pip install torch==2.0.0 --index-url https://download.pytorch.org/whl/cpu/
  Downloading https://download.pytorch.org/whl/cpu/torch-2.0.0%2Bcpu-cp39-cp39-linux_x86_64.whl (195.4 MB)

[msaroufim]

Hi @malfet we don't align torchserve releases with pytorch releases but if we're doing a release we try to upgrade pytorch version, see branches release_ for a mapping and the requirements folder to find the corresponding version

Regarding installation we can dig at the bug but here's how it works at a high level how this should work: This https://github.com/pytorch/serve/blob/master/ts_scripts/install_dependencies.py#L43 should call this https://github.com/pytorch/serve/blob/master/requirements/torch_linux.txt#L2 so installing GPU binaries on CPU is not expected

Regarding image size increases I'm personally more worried about GPU, we did this preliminary study here pytorch/builder#1371 (comment) but @agunapal is gonna take a deeper look, no action is needed at this moment

[msaroufim]

Just ran python install_dependencies.py (no GPU config) on a fresh machine with a GPU on it and I correctly got the CPU binary

Collecting torch==2.0.0+cpu (from -r requirements/torch_linux.txt (line 4))                                                                                     
  Downloading https://download.pytorch.org/whl/cpu/torch-2.0.0%2Bcpu-cp38-cp38-linux_x86_64.whl (195.4 MB)  

And pip list shows torch 2.0.0+cpu

We run the same command in our Dockerfile https://github.com/pytorch/serve/blob/master/docker/Dockerfile#L81

but will mark this as high pri until we find the bug

[agunapal]

@RobinFrcd I would appreciate your feedback on the attached PR. I am curious if the slim base image works for your usecase.