MatGPTQ: Efficient and Accurate Inference over Nested Quantized Models
Official implementation of MatGPTQ (Matryoshka GPTQ), a new PTQ pipeline that produces a single parent model jointly optimized for multiple target precisions in one-shot, based on a small calibration set.
- [2026/10] Released MatGPTQ v2 with improved slicing and faster kernels.
- [2026/02] Initial release of MatGPTQ.
Matryoshka Quantization (MatQuant), Any-Precision-LLM (AP) and AnyBCQ (AB) are recent quantization approaches showing that a single integer-quantized model can be served across multiple precisions. In this paradigm, lower-precision models are extracted from a higher-precision model by simply reading fewer bits of the weights. This enables a single checkpoint to cover a wide range of memory and latency budgets, but makes both quantization and efficient execution substantially harder. Existing methods rely on expensive quantization-aware training (QAT) or gradient-based post-training quantization (PTQ) rather than fast one-shot PTQ, and offer limited system support: dedicated kernels are either missing or restricted to single- or small-batch decoding. We address these limitations with Post-Training Matryoshka Quantization (MatGPTQ), an end-to-end pipeline for nested-model quantization and inference. MatGPTQ casts Matryoshka quantization as multi-precision error compensation, producing a single "sliceable" parent model jointly optimized for multiple target precisions in one pass over a small calibration set. We further refine the MatQuant representation so that an
scripts/— contains bash scripts with the required arguments to run the methodsrc/— directory for helper methods and utility functionsinference_lib/— directory contains cuda-kernel and deployment, packing and benchmarking scriptspaper/— scripts for the kernel and slicing figures of the paperquant.py— MatGPTQ/GPTQ quantizationlmeval.py— LM Eval Harness evalution scriptlmeval_vllm.py— LM Eval Harness evalution script for packed models in vLLMeval_ppl.py— perplexity evalution scriptsetup.py- installation script for cuda-kernelsrequirements.txt- requirements file
Create a virtual environment and install dependencies (we recommend Python 3.12):
uv venv --python 3.12
source .venv/bin/activate
uv pip install -r requirements.txtNote: The code has been tested with CUDA 12.4 and PyTorch 2.7.1
We provide quant.py for producing the MatGPTQ/GPTQ models. To produce the respective model see either scripts/run_gptq.sh or scripts/run_matgptq.sh for examples on how to run quantized training:
bash scripts/run_matgptq.sh--method—matgptqorgptq--bitwidth— the GPTQ bitwidth--bitwidth_options,--bitwidth_weights,--master_bitwidth— the MatGPTQ bitwidths optimized jointly, one weight per bitwidth (in the same order), and the largest of them (at most 8), whose codes are stored--scale_search— fit each group's scale and zero point to all bitwidths instead of the master's weight range (--scale_search_*tune it)--group_size,--perchannel,--sym,--act_order,--rel_damp,--block_size— the usual GPTQ settings--calibration_data,--calibration_tokens,--calibration_sequence_length— the calibration set (c4,fineweb_edu,wikitext2), its size and sequence length--quantizable_modules,--pre_block_modules,--block_modules,--post_block_modules— the layers to quantize (a regex) and the modules before, of and after the transformer blocks--save_dir— the quantized weights are written to<save_dir>/<model name>/<layer>/(data.ptfor MatGPTQ,<bitwidth>.ptfor GPTQ)--seed— seeds the calibration draw (the dataset shuffle). The paper reports five draws per model, seeds 0, 42, 1024, 2026 and 9999, and the standard deviations over them--mse— GPTQ only: pick each group's(scale, zero)by a per-row grid search over shrunk ranges, minimisingsum |w - Q(w)|^norm, instead of the plain min/max fit (--mse_grid,--mse_maxshrink,--mse_normtune it). The dedicated GPTQ baseline in the paper is GPTQ+MSE, soscripts/run_gptq.shpasses it
We provide lmeval.py and eval_ppl.py scripts for evaluation on the Language Model Evaluation Harness benchmarks and perplexity measurements. The interface of lmeval.py mostly follows the instructions from the original. In addition, one should specify the path to quantized weights via the quant_weights_path argument and the default uniform quantization bitwidth quant_uniform_bitwidth and master bitwidth --quant_master_bitwidth, or a path to a .txt file with chosen compression levels via the --quant_non_uniform_config_path argument. Furthermore, with --method, you define whether to evaluate MatGPTQ or GPTQ.
To evaluate packed models (see Packing) with our vLLM plugin, use lmeval_vllm.py with --model vllm and set the bitwidth with the SLICE_LEVEL environment variable.
scripts/run_lmeval_vllm.sh and scripts/run_lmeval.sh reproduce the evaluation used in the paper. The suite is split across two calls per model because the generative tasks need the chat template, and the reported average weights four groups equally:
| group | tasks | settings |
|---|---|---|
| CSR | ARC-Challenge, ARC-Easy, HellaSwag, PIQA, WinoGrande | 0-shot, log-likelihood |
| GSM8K | gsm8k_llama (Llama) / gsm8k (Qwen) |
generative, exact match |
| MMLU | mmlu_llama (Llama) / mmlu 5-shot (Qwen) |
— |
| IFEval | ifeval |
mean of prompt- and instance-level strict accuracy |
Avg is mean(CSR, GSM8K, MMLU, IFEval), so instruction following stays at one quarter of the total rather than letting two correlated IFEval metrics take two fifths.
Harness parameters: --seed 1234, --batch_size auto, and --apply_chat_template --fewshot_as_multiturn for the generative group. Serving parameters: dtype=bfloat16, max_model_len=4096, gpu_memory_utilization=0.85, attention_backend=FLASH_ATTN. On Qwen3, enable_thinking=False so the generative tasks are scored on the answer rather than on a chain of thought.
For deployment install our custom kernels and use our vLLM plugin to run it. Further information can be found in ./inference_lib.
You can start by using our already quantized models at ISTA-DASLab/MatGPTQ. Simply set the SLICE_LEVEL environment variable to your desired bitwidth (2, 3, 4, 6 or 8), and you're all set.
To install the kernel run the following command:
uv pip install --no-build-isolation -e . To deploy MatGPTQ models into various environment, you can use MatGPTQ with our vLLM plugin as followed (more information here). We recommend to use a clean and new environment (tested with vllm==0.27.1):
import vllm_matgptq
from vllm import LLM
llm = LLM(model="ISTA-DASLab/Llama-3.1-8B-Instruct-MatGPTQ", quantization="matgptq_quant")See inference_lib/inference_demo_vllm.py as an example on how to implement it. To run the model see inference_lib/scripts/run_inference_vllm.sh for an example.
Models can be packed using inference_lib/pack_matgptq_for_vllm.py as followed:
python inference_lib/pack_matgptq_for_vllm.py \
--base /path/to/model \
--quantized /path/to/quantized/weights \
--out /output/path \
--default_slice 4 \
--master_bitwidth 8 \
--group_size 128If you use MatGPTQ in your research, please cite:
@misc{kleinegger2026matgptqefficientaccurateinference,
title={MatGPTQ: Efficient and Accurate Inference over Nested Quantized Models},
author={Maximilian Kleinegger and Elvir Crnčević and Dan Alistarh},
year={2026},
eprint={2602.03537},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2602.03537},
}