[Bug] [qwen2/2.5-7b-instruct] VLLM加载模型失败

[xuanxuanzl]

Search before asking

• I had searched in the issues and found no similar issues.

Operating system information

Linux

Python version information

3.10

DB-GPT version

main

Related scenes

• Chat Data
• Chat Excel
• Chat DB
• Chat Knowledge
• Model Management
• Dashboard
• Plugins

Installation Information

• Installation From Source

• Docker Installation

• Docker Compose Installation

• Cluster Installation

• AutoDL Image

• Other

Device information

GPU RTX 4090*2

Models information

LLM_MODEL=qwen2.5-7b-instruct
EMBEDDING_MODEL=bge-m3

What happened

正常加载

What you expected to happen

0

How to reproduce

0

Additional context

RERAN2025-02-10 16:13:43 DESKTOP-E9OPQK4 dbgpt.model.adapter.vllm_adapter[48572] INFO Start vllm AsyncLLMEngine with args:

=========================== VLLMModelParameters ===========================

task: auto
tokenizer: None
skip_tokenizer_init: False
revision: None
code_revision: None
tokenizer_revision: None
tokenizer_mode: auto
trust_remote_code: True
allowed_local_media_path: None
download_dir: None
load_format: auto
config_format: ConfigFormat.AUTO
dtype: auto
kv_cache_dtype: auto
max_model_len: None
guided_decoding_backend: xgrammar
logits_processor_pattern: None
model_impl: auto
distributed_executor_backend: None
pipeline_parallel_size: 1
tensor_parallel_size: 2
max_parallel_loading_workers: None
ray_workers_use_nsight: False
block_size: None
enable_prefix_caching: None
disable_sliding_window: False
use_v2_block_manager: True
num_lookahead_slots: 0
seed: 0
swap_space: 4
cpu_offload_gb: 0
gpu_memory_utilization: 0.9
num_gpu_blocks_override: None
max_num_batched_tokens: None
max_num_seqs: None
max_logprobs: 20
disable_log_stats: False
quantization: None
rope_scaling: None
rope_theta: None
hf_overrides: None
enforce_eager: False
max_seq_len_to_capture: 8192
disable_custom_all_reduce: False
tokenizer_pool_size: 0
tokenizer_pool_type: ray
tokenizer_pool_extra_config: None
limit_mm_per_prompt: None
mm_processor_kwargs: None
disable_mm_preprocessor_cache: False
enable_lora: False
enable_lora_bias: False
max_loras: 1
max_lora_rank: 16
lora_extra_vocab_size: 256
lora_dtype: auto
long_lora_scaling_factors: None
max_cpu_loras: None
fully_sharded_loras: False
enable_prompt_adapter: False
max_prompt_adapters: 1
max_prompt_adapter_token: 0
device: auto
num_scheduler_steps: 1
multi_step_stream_outputs: True
scheduler_delay_factor: 0.0
enable_chunked_prefill: None
speculative_model: None
speculative_model_quantization: None
num_speculative_tokens: None
speculative_disable_mqa_scorer: False
speculative_draft_tensor_parallel_size: None
speculative_max_model_len: None
speculative_disable_by_batch_size: None
ngram_prompt_lookup_max: None
ngram_prompt_lookup_min: None
spec_decoding_acceptance_method: rejection_sampler
typical_acceptance_sampler_posterior_threshold: None
typical_acceptance_sampler_posterior_alpha: None
disable_logprobs_during_spec_decoding: None
model_loader_extra_config: None
ignore_patterns: None
preemption_mode: None
served_model_name: None
qlora_adapter_name_or_path: None
otlp_traces_endpoint: None
collect_detailed_traces: None
disable_async_output_proc: False
scheduling_policy: fcfs
override_neuron_config: None
override_pooler_config: None
compilation_config: None
kv_transfer_config: None
worker_cls: auto
generation_config: None
override_generation_config: None
enable_sleep_mode: False
calculate_kv_scales: False
disable_log_requests: False
model_name: qwen2.5-7b-instruct
model_path: /mnt/d/project/DB-GPT-main/models/Qwen2.5-7B-Instruct
model_type: vllm
prompt_template: None

======================================================================

2025-02-10 16:13:43 DESKTOP-E9OPQK4 dbgpt.model.cluster.worker.manager[48572] ERROR Error starting worker manager: model qwen2.5-7b-instruct@vllm(172.17.86.174:5670) start failed, Traceback (most recent call last):
File "/mnt/d/project/DB-GPT-main/dbgpt/model/cluster/worker/manager.py", line 507, in _start_worker
await self.run_blocking_func(
File "/mnt/d/project/DB-GPT-main/dbgpt/model/cluster/worker/manager.py", line 105, in run_blocking_func
return await loop.run_in_executor(self.executor, func, *args)
File "/home/lei/anaconda3/envs/dbgpt/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/mnt/d/project/DB-GPT-main/dbgpt/model/cluster/worker/default_worker.py", line 117, in start
self.model, self.tokenizer = self.ml.loader_with_params(
File "/mnt/d/project/DB-GPT-main/dbgpt/model/adapter/loader.py", line 132, in loader_with_params
return llm_adapter.load_from_params(model_params)
File "/mnt/d/project/DB-GPT-main/dbgpt/model/adapter/vllm_adapter.py", line 78, in load_from_params
engine = AsyncLLMEngine.from_engine_args(engine_args)
File "/home/lei/anaconda3/envs/dbgpt/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 639, in from_engine_args
engine_config = engine_args.create_engine_config(usage_context)
File "/home/lei/anaconda3/envs/dbgpt/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 1075, in create_engine_config
model_config = self.create_model_config()
File "/home/lei/anaconda3/envs/dbgpt/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 998, in create_model_config
return ModelConfig(
File "/home/lei/anaconda3/envs/dbgpt/lib/python3.10/site-packages/vllm/config.py", line 302, in init
hf_config = get_config(self.model, trust_remote_code, revision,
File "/home/lei/anaconda3/envs/dbgpt/lib/python3.10/site-packages/vllm/transformers_utils/config.py", line 250, in get_config
raise ValueError(f"Unsupported config format: {config_format}")
ValueError: Unsupported config format: ConfigFormat.AUTO

INFO: Shutting down
INFO: Waiting for application shutdown.
2025-02-10 16:13:43 DESKTOP-E9OPQK4 dbgpt.model.cluster.worker.manager[48572] INFO Stop all workers
2025-02-10 16:13:43 DESKTOP-E9OPQK4 dbgpt.model.cluster.worker.manager[48572] INFO Apply req: None, apply_func: <function LocalWorkerManager._stop_all_worker.._stop_worker at 0x7f6ef8a409d0>
2025-02-10 16:13:43 DESKTOP-E9OPQK4 dbgpt.model.cluster.worker.manager[48572] INFO Apply to all workers
2025-02-10 16:13:43 DESKTOP-E9OPQK4 dbgpt.model.cluster.worker.default_worker[48572] WARNING Model has been stopped!!
2025-02-10 16:13:45 DESKTOP-E9OPQK4 dbgpt.util.code.server[48572] INFO Code server is ready
2025-02-10 16:13:45 DESKTOP-E9OPQK4 dbgpt.util.api_utils[48572] WARNING No healthy urls found, selecting randomly
2025-02-10 16:13:45 DESKTOP-E9OPQK4 dbgpt.util.api_utils[48572] WARNING No healthy urls found, selecting randomly
2025-02-10 16:13:45 DESKTOP-E9OPQK4 dbgpt.model.cluster.worker.manager[48572] WARNING Stop worker, ignored exception from deregister_func: All connection attempts failed
2025-02-10 16:13:45 DESKTOP-E9OPQK4 dbgpt.model.cluster.worker.manager[48572] WARNING Stop worker, ignored exception from deregister_func: All connection attempts failed
INFO: Application shutdown complete.
INFO: Finished server process [48572]
in order to avoid chroma db atexit problemK_MODEL=bge-reranker-v2-m3

Are you willing to submit PR?

• Yes I am willing to submit a PR!