Token usage always shows 0.00

[Desmond819]

Hi, is there any reason the token usage is always 0.00? I keep sending request but the usage never go up.

Environment:
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA A100 80GB PCIe
GPU 0,1,2,3,4,5,6,7 Compute Capability: 8.0
CUDA_HOME: /usr/local/cuda-12.5
NVCC: Cuda compilation tools, release 12.5, V12.5.82
CUDA Driver Version: 560.28.03
PyTorch: 2.4.0+cu121
sglang: 0.2.13
flashinfer: 0.1.5+cu121torch2.4
triton: 3.0.0
transformers: 4.44.0
requests: 2.32.3
tqdm: 4.66.5
numpy: 1.26.4
aiohttp: 3.10.3
fastapi: 0.112.1
hf_transfer: 0.1.8
huggingface_hub: 0.24.5
interegular: 0.3.3
packaging: 24.1
PIL: 10.4.0
psutil: 6.0.0
pydantic: 2.8.2
uvicorn: 0.30.6
uvloop: 0.20.0
zmq: 26.1.0
vllm: 0.5.4
multipart: 0.0.9
openai: 1.40.8
anthropic: 0.34.0

[Desmond819]

I manually update the logger. Here is some additional information. Does this look alright?

[gpu=0] Decode batch. #running-req: 1, #token: 665, total tokens: 4144976, kv pool tokens: 4016566, tree cache: 127745, token usage: 0.00, gen throughput (token/s): 178.62, #queue-req: 0

    logger.info(
        f"[gpu={self.gpu_id}] Decode batch. "
        f"#running-req: {len(self.running_batch.reqs)}, "
        f"#token: {num_used}, "
        f"total tokens: {self.max_total_num_tokens}, " 
        f"kv pool tokens: {self.token_to_kv_pool.available_size()}, "  
        f"tree cache: {self.tree_cache.evictable_size()}, "                                   
        f"token usage: {num_used / self.max_total_num_tokens:.2f}, "
        f"gen throughput (token/s): {throughput:.2f}, "
        f"#queue-req: {len(self.waiting_queue)}"
    )

[zhyncs]

cc @hnyls2002

[Desmond819]

anyone knows? and anything I can check?

[merrymercy]

You submit the requests too slow. Use parallel threads to send the requests.