[bug] tgi-1.1.0 - Please install EETQ from https://github.com/NetEase-FuXi/EETQ

[Daan-Grashoff]

Checklist

• I've prepended issue tag with type of change: [bug]
• (If applicable) I've attached the script to reproduce the bug
• (If applicable) I've documented below the DLC image/dockerfile this relates to
• (If applicable) I've documented below the tests I've run on the DLC image
• I'm using an existing DLC image listed here: https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/deep-learning-containers-images.html
• I've built my own container based off DLC (and I've attached the code used to build my own image)

Concise Description:
When deploying a model with EETQ HF_MODEL_QUANTIZE, it results in the error "ImportError: Please install EETQ from https://github.com/NetEase-FuXi/EETQ".

DLC image/dockerfile:

• v1.0-hf-tgi-1.1.0-pt-2.0.1-inf-gpu-py39
• 763104351884.dkr.ecr.us-east-2.amazonaws.com/huggingface-pytorch-tgi-inference:2.0.1-tgi1.1.0-gpu-py39-cu118-ubuntu20.04-v1.0

Current behavior:
Returns with error ImportError: Please install EETQ from https://github.com/NetEase-FuXi/EETQ

Expected behavior:
The model should deploy and run without any issues.

Additional context:

[chintanckg]

I am facing the same issue!

[Daan-Grashoff]

@chintanckg Which image are you using?
763104351884.dkr.ecr.us-east-2.amazonaws.com/huggingface-pytorch-tgi-inference:2.0.1-tgi1.1.0-gpu-py39-cu118-ubuntu20.04
or
763104351884.dkr.ecr.us-east-2.amazonaws.com/huggingface-pytorch-tgi-inference:2.0.1-tgi1.1.0-gpu-py39-cu118-ubuntu20.04-v1.0

[chintanckg]

I am not sure how to get the exact image version, please help me with it.

[chintanckg]

I used the :latest tag and all is sorted now.

[Daan-Grashoff]

Can you share your code?

[chintanckg]

model= #path to model or hugging face path

volume=$PWD

docker run --gpus all --shm-size 24g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:latest --model-id $model --max-total-tokens 5024 --max-input-length 4096 --num-shard 4 --max-concurrent-requests 128 --quantize eetq