cutlass_scaled_mm RuntimeError: Expected a.dim() == 2 && b.dim() == 2 && c.dim() == 2 in quantized linear layer (Mistral 3.1 + vLLM + CUTLASS kernel)

[Alex31840]

While using a quantized mistral small 3.1 model with vLLM, I encountered a fatal runtime error during execution. The stack trace indicates the error originates from the custom CUTLASS kernel used for quantized matrix multiplication:
RuntimeError: Expected a.dim() == 2 && b.dim() == 2 && c.dim() == 2 to be true, but got false.

This occurs in cutlass_scaled_mm during self.qkv_proj(hidden_states). The root issue seems to be caused by one or more tensors not being 2D as expected by the quantization kernel.

System context:
Model: mistralai/Mistral-Small-3.1-24B-Instruct-2503
Quantization: compressed_tensors_w8a8_int8
Library: PyTorch 2.7
Environment: CUDA/NCCL backend, AWS p4d.24xlarge, llm-compressor (main branch), vLLM(0.9.1)

This issue consistently occurs across multiple implementation attempts and configurations using llm-compressor.

Recipe:
if scheme == "W8A8":
return [
SmoothQuantModifier(
smoothing_strength=0.8,
mappings=[
[["re:^(?!.*vision_tower).*q_proj", "re:^(?!.*vision_tower).*k_proj", "re:^(?!.*vision_tower).*v_proj"], "re:^(?!.*vision_tower).*input_layernorm"],
[["re:^(?!.*vision_tower).*gate_proj", "re:^(?!.*vision_tower).*up_proj"], "re:^(?!.*vision_tower).*post_attention_layernorm"],
[["re:^(?!.*vision_tower).*down_proj"], "re:^(?!.*vision_tower).*up_proj"],
],
),
GPTQModifier(
ignore=[
"lm_head",
"re:.vision_tower.",
"re:.multi_modal_projector."
],
sequential_targets=["MistralDecoderLayer"],
dampening_frac=0.01,
targets="Linear",
scheme=scheme,
# offload_hessians=True
),
]

MAX_SEQ_LEN = 8192

I've also tested with Mistral3ForConditionalGeneration
model = AutoModelForImageTextToText.from_pretrained(
MODEL_ID,
device_map="balanced",
torch_dtype="auto",
)

processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)

oneshot(
model=model,
dataset=dataset,
recipe=recipe,
max_seq_length=MAX_SEQ_LEN,
data_collator=create_data_collator,
num_calibration_samples=len(dataset),
trust_remote_code_model=True,
)

save_path = f"{model_name}-quantized.{scheme.lower()}"
logger.info("Saving quantized model to: %s", save_path)

os.makedirs(save_path, exist_ok=True)
model.save_pretrained(save_path, save_compressed=True)
processor.save_pretrained(save_path)

The original ignore filter was not respected by GPTQModifier; it seems the relevant modules (e.g., vision tower, ...) were not found or filtered as expected.

INFO 06-13 08:42:50 [loader.py:447] Loading weights took 6.95 seconds
INFO 06-13 08:42:51 [gpu_model_runner.py:1273] Model loading took 23.6805 GiB and 7.212285 seconds
INFO 06-13 08:42:51 [gpu_model_runner.py:1542] Encoder cache will be initialized with a budget of 3080 tokens, and profiled with 1 image items of the maximum feature size.
ERROR 06-13 08:42:53 [core.py:390] EngineCore hit an exception: Traceback (most recent call last):
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 378, in run_engine_core
ERROR 06-13 08:42:53 [core.py:390] engine_core = EngineCoreProc(*args, **kwargs)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 319, in init
ERROR 06-13 08:42:53 [core.py:390] super().init(vllm_config, executor_class, log_stats)
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 71, in init
ERROR 06-13 08:42:53 [core.py:390] self._initialize_kv_caches(vllm_config)
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 132, in _initialize_kv_caches
ERROR 06-13 08:42:53 [core.py:390] available_gpu_memory = self.model_executor.determine_available_memory()
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 66, in determine_available_memory
ERROR 06-13 08:42:53 [core.py:390] output = self.collective_rpc("determine_available_memory")
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 06-13 08:42:53 [core.py:390] answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/utils.py", line 2347, in run_method
ERROR 06-13 08:42:53 [core.py:390] return func(*args, **kwargs)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 06-13 08:42:53 [core.py:390] return func(*args, **kwargs)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 157, in determine_available_memory
ERROR 06-13 08:42:53 [core.py:390] self.model_runner.profile_run()
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1562, in profile_run
ERROR 06-13 08:42:53 [core.py:390] dummy_encoder_outputs = self.model.get_multimodal_embeddings(
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/model_executor/models/mistral3.py", line 558, in get_multimodal_embeddings
ERROR 06-13 08:42:53 [core.py:390] vision_embeddings = self._process_image_input(image_input)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/model_executor/models/mistral3.py", line 534, in _process_image_input
ERROR 06-13 08:42:53 [core.py:390] image_features = self.vision_tower(image_input["pixel_values"])
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
ERROR 06-13 08:42:53 [core.py:390] return self._call_impl(*args, **kwargs)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
ERROR 06-13 08:42:53 [core.py:390] return forward_call(*args, **kwargs)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/model_executor/models/pixtral.py", line 1300, in forward
ERROR 06-13 08:42:53 [core.py:390] out = self.transformer(
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
ERROR 06-13 08:42:53 [core.py:390] return self._call_impl(*args, **kwargs)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
ERROR 06-13 08:42:53 [core.py:390] return forward_call(*args, **kwargs)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/model_executor/models/pixtral.py", line 1191, in forward
ERROR 06-13 08:42:53 [core.py:390] x = layer(x, attention_mask, position_embeddings)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
ERROR 06-13 08:42:53 [core.py:390] return self._call_impl(*args, **kwargs)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
ERROR 06-13 08:42:53 [core.py:390] return forward_call(*args, **kwargs)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/model_executor/models/pixtral.py", line 1148, in forward
ERROR 06-13 08:42:53 [core.py:390] r, _ = self.attention.forward(self.attention_norm(hidden_states),
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/model_executor/models/pixtral.py", line 1091, in forward
ERROR 06-13 08:42:53 [core.py:390] qkv_states, _ = self.qkv_proj(hidden_states)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
ERROR 06-13 08:42:53 [core.py:390] return self._call_impl(*args, **kwargs)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in call_impl
ERROR 06-13 08:42:53 [core.py:390] return forward_call(*args, **kwargs)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/model_executor/layers/linear.py", line 474, in forward
ERROR 06-13 08:42:53 [core.py:390] output_parallel = self.quant_method.apply(self, input, bias)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors.py", line 581, in apply
ERROR 06-13 08:42:53 [core.py:390] return scheme.apply_weights(layer, x, bias=bias)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8.py", line 110, in apply_weights
ERROR 06-13 08:42:53 [core.py:390] return self.kernel.apply_weights(layer, x, bias)
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/kernels/scaled_mm/cutlass.py", line 131, in apply_weights
ERROR 06-13 08:42:53 [core.py:390] return ops.cutlass_scaled_mm(x_q,
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/vllm/_custom_ops.py", line 557, in cutlass_scaled_mm
ERROR 06-13 08:42:53 [core.py:390] torch.ops._C.cutlass_scaled_mm(out, a, b, scale_a, scale_b, bias)
ERROR 06-13 08:42:53 [core.py:390] File "/opt/pytorch/lib/python3.12/site-packages/torch/_ops.py", line 1123, in call
ERROR 06-13 08:42:53 [core.py:390] return self._op(*args, **(kwargs or {}))
ERROR 06-13 08:42:53 [core.py:390] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-13 08:42:53 [core.py:390] RuntimeError: Expected a.dim() == 2 && b.dim() == 2 && c.dim() == 2 to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.)
ERROR 06-13 08:42:53 [core.py:390]
CRITICAL 06-13 08:42:53 [core_client.py:361] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
Killed

[kylesayrs]

@Alex31840 Thanks for the catch! Huggingface recently changed their checkpoints formats, which has broken many of our ignore lists #1545. It seems that Mistral3 is one of those models

[Alex31840]

That's what I did. The following ignore matches the model.vision_tower.*

ignore=[
"lm_head",
"re:.*vision_tower.*",
"re:.*multi_modal_projector.*"
],

Even with a config.json that appears correct, I get this error -> RuntimeError: Expected a.dim() == 2 && b.dim() == 2 && c.dim() == 2 to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.) :/

[Alex31840]

Hi,

I got a new error on the current branch.

vllm serve Mistral-Small-3.1-24B-Instruct-2503-quantized.w8a8 --tool-call-parser mistral --enable-auto-tool-choice --limit_mm_per_prompt 'image=3' --tensor-parallel-size 1

INFO 06-21 09:57:36 [gpu_model_runner.py:1595] Starting to load model Mistral-Small-3.1-24B-Instruct-2503-quantized.w8a8...
INFO 06-21 09:57:36 [gpu_model_runner.py:1600] Loading model from scratch...
INFO 06-21 09:57:36 [compressed_tensors_w8a8_int8.py:52] Using CutlassScaledMMLinearKernel for CompressedTensorsW8A8Int8
INFO 06-21 09:57:36 [cuda.py:252] Using Flash Attention backend on V1 engine.
Loading safetensors checkpoint shards: 0% Completed | 0/6 [00:00<?, ?it/s]
ERROR 06-21 09:57:37 [core.py:515] EngineCore failed to start.
ERROR 06-21 09:57:37 [core.py:515] Traceback (most recent call last):
ERROR 06-21 09:57:37 [core.py:515] File "/opt/pytorch/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 506, in run_engine_core
ERROR 06-21 09:57:37 [core.py:515] engine_core = EngineCoreProc(*args, **kwargs)
ERROR 06-21 09:57:37 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-21 09:57:37 [core.py:515] File "/opt/pytorch/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 390, in init
ERROR 06-21 09:57:37 [core.py:515] super().init(vllm_config, executor_class, log_stats,
ERROR 06-21 09:57:37 [core.py:515] File "/opt/pytorch/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 76, in init
ERROR 06-21 09:57:37 [core.py:515] self.model_executor = executor_class(vllm_config)
ERROR 06-21 09:57:37 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-21 09:57:37 [core.py:515] File "/opt/pytorch/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 53, in init
ERROR 06-21 09:57:37 [core.py:515] self._init_executor()
ERROR 06-21 09:57:37 [core.py:515] File "/opt/pytorch/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 48, in _init_executor
ERROR 06-21 09:57:37 [core.py:515] self.collective_rpc("load_model")
ERROR 06-21 09:57:37 [core.py:515] File "/opt/pytorch/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 57, in collective_rpc
ERROR 06-21 09:57:37 [core.py:515] answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 06-21 09:57:37 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-21 09:57:37 [core.py:515] File "/opt/pytorch/lib/python3.12/site-packages/vllm/utils.py", line 2671, in run_method
ERROR 06-21 09:57:37 [core.py:515] return func(*args, **kwargs)
ERROR 06-21 09:57:37 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^
ERROR 06-21 09:57:37 [core.py:515] File "/opt/pytorch/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 180, in load_model
ERROR 06-21 09:57:37 [core.py:515] self.model_runner.load_model()
ERROR 06-21 09:57:37 [core.py:515] File "/opt/pytorch/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1601, in load_model
ERROR 06-21 09:57:37 [core.py:515] self.model = model_loader.load_model(
ERROR 06-21 09:57:37 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-21 09:57:37 [core.py:515] File "/opt/pytorch/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 41, in load_model
ERROR 06-21 09:57:37 [core.py:515] self.load_weights(model, model_config)
ERROR 06-21 09:57:37 [core.py:515] File "/opt/pytorch/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 269, in load_weights
ERROR 06-21 09:57:37 [core.py:515] loaded_weights = model.load_weights(
ERROR 06-21 09:57:37 [core.py:515] ^^^^^^^^^^^^^^^^^^^
ERROR 06-21 09:57:37 [core.py:515] File "/opt/pytorch/lib/python3.12/site-packages/vllm/model_executor/models/mistral3.py", line 595, in load_weights
ERROR 06-21 09:57:37 [core.py:515] return loader.load_weights(weights)
ERROR 06-21 09:57:37 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-21 09:57:37 [core.py:515] File "/opt/pytorch/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 278, in load_weights
ERROR 06-21 09:57:37 [core.py:515] autoloaded_weights = set(self._load_module("", self.module, weights))
ERROR 06-21 09:57:37 [core.py:515] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-21 09:57:37 [core.py:515] File "/opt/pytorch/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 264, in _load_module
ERROR 06-21 09:57:37 [core.py:515] raise ValueError(msg)
ERROR 06-21 09:57:37 [core.py:515] ValueError: There is no module or parameter named 'model' in Mistral3ForConditionalGeneration

My recipe =>
default_stage:
default_modifiers:
SmoothQuantModifier:
smoothing_strength: 0.8
mappings:
- - ['re:^(?!.*vision_tower).*q_proj', 're:^(?!.*vision_tower).*k_proj', 're:^(?!.*vision_tower).*v_proj']
- re:^(?!.*vision_tower).*input_layernorm
- - ['re:^(?!.*vision_tower).*gate_proj', 're:^(?!.*vision_tower).*up_proj']
- re:^(?!.*vision_tower).*post_attention_layernorm
- - ['re:^(?!.*vision_tower).*down_proj']
- re:^(?!.*vision_tower).*up_proj
GPTQModifier:
targets: [Linear]
ignore: ['re:.*lm_head', 're:.*vision_tower.*', 're:.*multi_modal_projector.*']
scheme: W8A8
sequential_targets: [MistralDecoderLayer]
dampening_frac: 0.01

My config.json =>
{
"architectures": [
"Mistral3ForConditionalGeneration"
],
"image_token_index": 10,
"model_type": "mistral3",
"multimodal_projector_bias": false,
"projector_hidden_act": "gelu",
"quantization_config": {
"config_groups": {
"group_0": {
"input_activations": {
"actorder": null,
"block_structure": null,
"dynamic": true,
"group_size": null,
"num_bits": 8,
"observer": null,
"observer_kwargs": {},
"strategy": "token",
"symmetric": true,
"type": "int"
},
"output_activations": null,
"targets": [
"Linear"
],
"weights": {
"actorder": null,
"block_structure": null,
"dynamic": false,
"group_size": null,
"num_bits": 8,
"observer": "minmax",
"observer_kwargs": {},
"strategy": "channel",
"symmetric": true,
"type": "int"
}
}
},
"format": "int-quantized",
"global_compression_ratio": null,
"ignore": [
"model.vision_tower.transformer.layers.0.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.0.feed_forward.up_proj",
"model.vision_tower.transformer.layers.0.feed_forward.down_proj",
"model.vision_tower.transformer.layers.0.attention.k_proj",
"model.vision_tower.transformer.layers.0.attention.v_proj",
"model.vision_tower.transformer.layers.0.attention.q_proj",
"model.vision_tower.transformer.layers.0.attention.o_proj",
"model.vision_tower.transformer.layers.1.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.1.feed_forward.up_proj",
"model.vision_tower.transformer.layers.1.feed_forward.down_proj",
"model.vision_tower.transformer.layers.1.attention.k_proj",
"model.vision_tower.transformer.layers.1.attention.v_proj",
"model.vision_tower.transformer.layers.1.attention.q_proj",
"model.vision_tower.transformer.layers.1.attention.o_proj",
"model.vision_tower.transformer.layers.2.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.2.feed_forward.up_proj",
"model.vision_tower.transformer.layers.2.feed_forward.down_proj",
"model.vision_tower.transformer.layers.2.attention.k_proj",
"model.vision_tower.transformer.layers.2.attention.v_proj",
"model.vision_tower.transformer.layers.2.attention.q_proj",
"model.vision_tower.transformer.layers.2.attention.o_proj",
"model.vision_tower.transformer.layers.3.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.3.feed_forward.up_proj",
"model.vision_tower.transformer.layers.3.feed_forward.down_proj",
"model.vision_tower.transformer.layers.3.attention.k_proj",
"model.vision_tower.transformer.layers.3.attention.v_proj",
"model.vision_tower.transformer.layers.3.attention.q_proj",
"model.vision_tower.transformer.layers.3.attention.o_proj",
"model.vision_tower.transformer.layers.4.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.4.feed_forward.up_proj",
"model.vision_tower.transformer.layers.4.feed_forward.down_proj",
"model.vision_tower.transformer.layers.4.attention.k_proj",
"model.vision_tower.transformer.layers.4.attention.v_proj",
"model.vision_tower.transformer.layers.4.attention.q_proj",
"model.vision_tower.transformer.layers.4.attention.o_proj",
"model.vision_tower.transformer.layers.5.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.5.feed_forward.up_proj",
"model.vision_tower.transformer.layers.5.feed_forward.down_proj",
"model.vision_tower.transformer.layers.5.attention.k_proj",
"model.vision_tower.transformer.layers.5.attention.v_proj",
"model.vision_tower.transformer.layers.5.attention.q_proj",
"model.vision_tower.transformer.layers.5.attention.o_proj",
"model.vision_tower.transformer.layers.6.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.6.feed_forward.up_proj",
"model.vision_tower.transformer.layers.6.feed_forward.down_proj",
"model.vision_tower.transformer.layers.6.attention.k_proj",
"model.vision_tower.transformer.layers.6.attention.v_proj",
"model.vision_tower.transformer.layers.6.attention.q_proj",
"model.vision_tower.transformer.layers.6.attention.o_proj",
"model.vision_tower.transformer.layers.7.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.7.feed_forward.up_proj",
"model.vision_tower.transformer.layers.7.feed_forward.down_proj",
"model.vision_tower.transformer.layers.7.attention.k_proj",
"model.vision_tower.transformer.layers.7.attention.v_proj",
"model.vision_tower.transformer.layers.7.attention.q_proj",
"model.vision_tower.transformer.layers.7.attention.o_proj",
"model.vision_tower.transformer.layers.8.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.8.feed_forward.up_proj",
"model.vision_tower.transformer.layers.8.feed_forward.down_proj",
"model.vision_tower.transformer.layers.8.attention.k_proj",
"model.vision_tower.transformer.layers.8.attention.v_proj",
"model.vision_tower.transformer.layers.8.attention.q_proj",
"model.vision_tower.transformer.layers.8.attention.o_proj",
"model.vision_tower.transformer.layers.9.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.9.feed_forward.up_proj",
"model.vision_tower.transformer.layers.9.feed_forward.down_proj",
"model.vision_tower.transformer.layers.9.attention.k_proj",
"model.vision_tower.transformer.layers.9.attention.v_proj",
"model.vision_tower.transformer.layers.9.attention.q_proj",
"model.vision_tower.transformer.layers.9.attention.o_proj",
"model.vision_tower.transformer.layers.10.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.10.feed_forward.up_proj",
"model.vision_tower.transformer.layers.10.feed_forward.down_proj",
"model.vision_tower.transformer.layers.10.attention.k_proj",
"model.vision_tower.transformer.layers.10.attention.v_proj",
"model.vision_tower.transformer.layers.10.attention.q_proj",
"model.vision_tower.transformer.layers.10.attention.o_proj",
"model.vision_tower.transformer.layers.11.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.11.feed_forward.up_proj",
"model.vision_tower.transformer.layers.11.feed_forward.down_proj",
"model.vision_tower.transformer.layers.11.attention.k_proj",
"model.vision_tower.transformer.layers.11.attention.v_proj",
"model.vision_tower.transformer.layers.11.attention.q_proj",
"model.vision_tower.transformer.layers.11.attention.o_proj",
"model.vision_tower.transformer.layers.12.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.12.feed_forward.up_proj",
"model.vision_tower.transformer.layers.12.feed_forward.down_proj",
"model.vision_tower.transformer.layers.12.attention.k_proj",
"model.vision_tower.transformer.layers.12.attention.v_proj",
"model.vision_tower.transformer.layers.12.attention.q_proj",
"model.vision_tower.transformer.layers.12.attention.o_proj",
"model.vision_tower.transformer.layers.13.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.13.feed_forward.up_proj",
"model.vision_tower.transformer.layers.13.feed_forward.down_proj",
"model.vision_tower.transformer.layers.13.attention.k_proj",
"model.vision_tower.transformer.layers.13.attention.v_proj",
"model.vision_tower.transformer.layers.13.attention.q_proj",
"model.vision_tower.transformer.layers.13.attention.o_proj",
"model.vision_tower.transformer.layers.14.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.14.feed_forward.up_proj",
"model.vision_tower.transformer.layers.14.feed_forward.down_proj",
"model.vision_tower.transformer.layers.14.attention.k_proj",
"model.vision_tower.transformer.layers.14.attention.v_proj",
"model.vision_tower.transformer.layers.14.attention.q_proj",
"model.vision_tower.transformer.layers.14.attention.o_proj",
"model.vision_tower.transformer.layers.15.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.15.feed_forward.up_proj",
"model.vision_tower.transformer.layers.15.feed_forward.down_proj",
"model.vision_tower.transformer.layers.15.attention.k_proj",
"model.vision_tower.transformer.layers.15.attention.v_proj",
"model.vision_tower.transformer.layers.15.attention.q_proj",
"model.vision_tower.transformer.layers.15.attention.o_proj",
"model.vision_tower.transformer.layers.16.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.16.feed_forward.up_proj",
"model.vision_tower.transformer.layers.16.feed_forward.down_proj",
"model.vision_tower.transformer.layers.16.attention.k_proj",
"model.vision_tower.transformer.layers.16.attention.v_proj",
"model.vision_tower.transformer.layers.16.attention.q_proj",
"model.vision_tower.transformer.layers.16.attention.o_proj",
"model.vision_tower.transformer.layers.17.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.17.feed_forward.up_proj",
"model.vision_tower.transformer.layers.17.feed_forward.down_proj",
"model.vision_tower.transformer.layers.17.attention.k_proj",
"model.vision_tower.transformer.layers.17.attention.v_proj",
"model.vision_tower.transformer.layers.17.attention.q_proj",
"model.vision_tower.transformer.layers.17.attention.o_proj",
"model.vision_tower.transformer.layers.18.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.18.feed_forward.up_proj",
"model.vision_tower.transformer.layers.18.feed_forward.down_proj",
"model.vision_tower.transformer.layers.18.attention.k_proj",
"model.vision_tower.transformer.layers.18.attention.v_proj",
"model.vision_tower.transformer.layers.18.attention.q_proj",
"model.vision_tower.transformer.layers.18.attention.o_proj",
"model.vision_tower.transformer.layers.19.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.19.feed_forward.up_proj",
"model.vision_tower.transformer.layers.19.feed_forward.down_proj",
"model.vision_tower.transformer.layers.19.attention.k_proj",
"model.vision_tower.transformer.layers.19.attention.v_proj",
"model.vision_tower.transformer.layers.19.attention.q_proj",
"model.vision_tower.transformer.layers.19.attention.o_proj",
"model.vision_tower.transformer.layers.20.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.20.feed_forward.up_proj",
"model.vision_tower.transformer.layers.20.feed_forward.down_proj",
"model.vision_tower.transformer.layers.20.attention.k_proj",
"model.vision_tower.transformer.layers.20.attention.v_proj",
"model.vision_tower.transformer.layers.20.attention.q_proj",
"model.vision_tower.transformer.layers.20.attention.o_proj",
"model.vision_tower.transformer.layers.21.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.21.feed_forward.up_proj",
"model.vision_tower.transformer.layers.21.feed_forward.down_proj",
"model.vision_tower.transformer.layers.21.attention.k_proj",
"model.vision_tower.transformer.layers.21.attention.v_proj",
"model.vision_tower.transformer.layers.21.attention.q_proj",
"model.vision_tower.transformer.layers.21.attention.o_proj",
"model.vision_tower.transformer.layers.22.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.22.feed_forward.up_proj",
"model.vision_tower.transformer.layers.22.feed_forward.down_proj",
"model.vision_tower.transformer.layers.22.attention.k_proj",
"model.vision_tower.transformer.layers.22.attention.v_proj",
"model.vision_tower.transformer.layers.22.attention.q_proj",
"model.vision_tower.transformer.layers.22.attention.o_proj",
"model.vision_tower.transformer.layers.23.feed_forward.gate_proj",
"model.vision_tower.transformer.layers.23.feed_forward.up_proj",
"model.vision_tower.transformer.layers.23.feed_forward.down_proj",
"model.vision_tower.transformer.layers.23.attention.k_proj",
"model.vision_tower.transformer.layers.23.attention.v_proj",
"model.vision_tower.transformer.layers.23.attention.q_proj",
"model.vision_tower.transformer.layers.23.attention.o_proj",
"model.multi_modal_projector.patch_merger.merging_layer",
"model.multi_modal_projector.linear_1",
"model.multi_modal_projector.linear_2",
"lm_head"
],
"kv_cache_scheme": null,
"quant_method": "compressed-tensors",
"quantization_status": "compressed"
},
"spatial_merge_size": 2,
"text_config": {
"attention_dropout": 0.0,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 5120,
"initializer_range": 0.02,
"intermediate_size": 32768,
"max_position_embeddings": 131072,
"model_type": "mistral",
"num_attention_heads": 32,
"num_hidden_layers": 40,
"num_key_value_heads": 8,
"rms_norm_eps": 1e-05,
"rope_theta": 1000000000.0,
"sliding_window": null,
"torch_dtype": "bfloat16",
"use_cache": true,
"vocab_size": 131072
},
"torch_dtype": "bfloat16",
"transformers_version": "4.52.2",
"vision_config": {
"attention_dropout": 0.0,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 1024,
"image_size": 1540,
"initializer_range": 0.02,
"intermediate_size": 4096,
"model_type": "pixtral",
"num_attention_heads": 16,
"num_channels": 3,
"num_hidden_layers": 24,
"patch_size": 14,
"rope_theta": 10000.0,
"torch_dtype": "bfloat16"
},
"vision_feature_layer": -1
}

I removed my custom chat template and dataset to test with the default settings. But now, I can't even load the model.

Feel free to ask for any information.

Have a great day.

[Alex31840]

With the current transformers branch =>
2025-06-21T10:56:19.809904+0000 | from_modifiers | INFO - Creating recipe from modifiers
2025-06-21T10:56:22.510140+0000 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
2025-06-21T10:56:22.510322+0000 | IndependentPipeline | INFO - Inferred SequentialPipeline for SmoothQuantModifier
ERROR:main:Quantization failed: symbolically traced variables cannot be used as inputs to control flow
Traceback (most recent call last):
File "/home/ubuntu/quant/llm_compressor.py", line 306, in main

[Jeremy-J-J]

I've encountered the same error on Qwen2d5VL (W8A8_INT8). How fix it?