[Bug] ValueError: The memory capacity is unbalanced

[chuangzhidan]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

==========
== CUDA ==

CUDA Version 12.1.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

[22:07:11] server_args=ServerArgs(model_path='/workspace/model/Qwen2.5-72-int4', tokenizer_path='/workspace/model/Qwen2.5-72-int4', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', dtype='auto', kv_cache_dtype='auto', trust_remote_code=False, context_length=None, quantization='gptq_marlin', served_model_name='/workspace/model/Qwen2.5-72-int4', chat_template=None, is_embedding=False, host='0.0.0.0', port=30000, additional_ports=[30001, 30002, 30003, 30004], mem_fraction_static=0.9, max_running_requests=None, max_num_reqs=None, max_total_tokens=None, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='lpm', schedule_conservativeness=1.0, tp_size=2, stream_interval=1, random_seed=245707448, log_level='info', log_level_http=None, log_requests=False, show_time_cost=False, api_key=None, file_storage_pth='SGLang_storage', dp_size=1, load_balance_method='round_robin', disable_flashinfer=False, disable_flashinfer_sampling=False, disable_radix_cache=False, disable_regex_jump_forward=False, disable_cuda_graph=True, disable_cuda_graph_padding=False, disable_disk_cache=False, disable_custom_all_reduce=False, enable_mixed_chunk=False, enable_torch_compile=False, enable_p2p_check=False, enable_mla=False, triton_attention_reduce_in_fp32=False, nccl_init_addr=None, nnodes=1, node_rank=None)
[22:07:13 TP0] Init nccl begin.
[22:07:16 TP1] Init nccl begin.
[22:07:25 TP1] Exception in run_tp_server:
Traceback (most recent call last):
File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 885, in run_tp_server
model_server = ModelTpServer(
File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 99, in init
self.model_runner = ModelRunner(
File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 109, in init
min_per_gpu_memory = self.init_torch_distributed()
File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 158, in init_torch_distributed
raise ValueError(
ValueError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes.

Process Process-1:1:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 885, in run_tp_server
model_server = ModelTpServer(
File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 99, in init
self.model_runner = ModelRunner(
File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 109, in init
min_per_gpu_memory = self.init_torch_distributed()
File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 158, in init_torch_distributed
raise ValueError(
ValueError: The memory capacity is unbalanced

memory usage is listed bellowed(A800*2, 80G):
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A800 80GB PCIe On | 00000000:17:00.0 Off | 0 |
| N/A 42C P0 68W / 300W | 48857MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA A800 80GB PCIe On | 00000000:31:00.0 Off | 0 |
| N/A 41C P0 69W / 300W | 38316MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+

Reproduction

docker run --gpus all -it -p 30000:30000 -v ~/.cache/huggingface:/root/.cache/huggingface -v /media/data/xgp:/workspace -v /media/data/llm:/workspace/model --env "HF_TOKEN=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" --env CUDA_VISIBLE_DEVICES=0,1 --ipc=host lmsysorg/sglang:latest python3 -m sglang.launch_server --model-path /workspace/model/Qwen2.5-72-int4 --host 0.0.0.0 --port 30000 --quantization gptq_marlin --mem-fraction-static 0.9 --disable-cuda-graph --tp-size 2

Environment

Python: 3.12.4 | packaged by Anaconda, Inc. | (main, Jun 18 2024, 15:12:24) [GCC 11.2.0]
CUDA available: True
GPU 0,1: NVIDIA A800 80GB PCIe
GPU 0,1 Compute Capability: 8.0
CUDA_HOME: /usr
NVCC: Cuda compilation tools, release 11.5, V11.5.119
CUDA Driver Version: 560.35.03
PyTorch: 2.4.0+cu121
sglang: 0.3.0
flashinfer: Module Not Found
triton: 3.0.0
transformers: 4.44.2
requests: 2.32.3
tqdm: 4.66.5
numpy: 1.26.4
aiohttp: 3.10.5
fastapi: 0.113.0
hf_transfer: Module Not Found
huggingface_hub: 0.24.6
interegular: 0.3.3
packaging: 24.1
PIL: 10.4.0
psutil: 6.0.0
pydantic: 2.9.0
uvicorn: 0.30.6
uvloop: 0.20.0
zmq: 26.2.0
vllm: 0.6.0
multipart: Module Not Found
openai: 1.43.1
anthropic: Module Not Found
litellm: Module Not Found
NVIDIA Topology:
GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NODE 0-23,48-71 0 N/A
GPU1 NODE X 0-23,48-71 0 N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks