[docs] torch.compile usage guide

[youkaichao]

No description provided.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[youkaichao]

I also run it with llama 3.2 1B model:

    $ # running a 1B model on H100 with batch size 1, 21.29 seconds of compilation time, 13.7% improvement in latency

    $ python3 benchmarks/benchmark_latency.py --model meta-llama/Llama-3.2-1B --batch-size 1 --load-format dummy --num-scheduler-steps 16
    init engine (profile, create kv cache, warmup model) took 11.79 seconds
    Avg latency: 0.2771991847005362 seconds

    $ python3 benchmarks/benchmark_latency.py --model meta-llama/Llama-3.2-1B --batch-size 1 --load-format dummy --num-scheduler-steps 16 -O "{'level': 3, 'candidate_compile_sizes': [1]}"
    init engine (profile, create kv cache, warmup model) took 33.08 seconds
    Avg latency: 0.23920089063079406 seconds

    $ # running a 1B model on L4 with batch size 1, 42.0 seconds of compilation time, 4.0 % improvement in latency

    $ python3 benchmarks/benchmark_latency.py --model meta-llama/Llama-3.2-1B --batch-size 1 --load-format dummy --num-scheduler-steps 16
    init engine (profile, create kv cache, warmup model) took 20.32 seconds
    Avg latency: 1.526933370166671 seconds

    $ python3 benchmarks/benchmark_latency.py --model meta-llama/Llama-3.2-1B --batch-size 1 --load-format dummy --num-scheduler-steps 16 -O "{'level': 3, 'candidate_compile_sizes': [1]}"
    init engine (profile, create kv cache, warmup model) took 62.32 seconds
    Avg latency: 1.4660025673666648 seconds

[tlrmchlsmth]

The inductor graph compilation time is inclusive of the triton kernel compilation time, right? Could you clarify that if so?

[youkaichao]

I don't think so. Some triton kernels might have JIT compilation that is not included in the inductor compilation time.

[tlrmchlsmth]

Should we document how to enable/disable custom inductor passes?

[youkaichao]

repeat for llama 3.2 1B on H100, almost no improvement.

    $ # running an 1B model on H100 with various batch sizes, 39.79 seconds of compilation time, 0.5% improvement in throughput
    $
    $ # 1. Run the baseline setting
    $ python3 benchmarks/benchmark_throughput.py --input-len 256 --output-len 256 --model meta-llama/Llama-3.2-1B --load-format dummy --num-scheduler-steps 64
    init engine (profile, create kv cache, warmup model) took 13.14 seconds
    Throughput: 116.83 requests/s, 59814.48 total tokens/s, 29907.24 output tokens/s

    $ # 2. Run the same setting with profiling
    $ VLLM_LOG_BATCHSIZE_INTERVAL=1.0 python3 benchmarks/benchmark_throughput.py --input-len 256 --output-len 256 --model meta-llama/Llama-3.2-1B --num-scheduler-steps 64
    INFO 12-10 15:42:47 forward_context.py:58] Batchsize distribution (batchsize, count): [(256, 769), (232, 215), ...]

    $ # 3. The most common batch sizes are 256 and 232, so we can compile the model for these two batch sizes
    $ python3 benchmarks/benchmark_throughput.py --input-len 256 --output-len 256 --model meta-llama/Llama-3.2-1B --num-scheduler-steps 64 -O "{'level': 3, 'candidate_compile_sizes': [232, 256]}"
    init engine (profile, create kv cache, warmup model) took 52.93 seconds
    Throughput: 117.38 requests/s, 60100.50 total tokens/s, 30050.25 output tokens/s

repeat for llama 3.2 1B on L4 (it's even slower):

    $ # running an 1B model on L4 with various batch sizes, 58.5 seconds of compilation time, -6.0% improvement in throughput
    $
    $ # 1. Run the baseline setting
    $ python3 benchmarks/benchmark_throughput.py --input-len 256 --output-len 256 --model meta-llama/Llama-3.2-1B --load-format dummy --num-scheduler-steps 64
    init engine (profile, create kv cache, warmup model) took 21.77 seconds
    Throughput: 16.36 requests/s, 8376.21 total tokens/s, 4188.10 output tokens/s

    $ # 2. Run the same setting with profiling
    $ VLLM_LOG_BATCHSIZE_INTERVAL=1.0 python3 benchmarks/benchmark_throughput.py --input-len 256 --output-len 256 --model meta-llama/Llama-3.2-1B --load-format dummy --num-scheduler-steps 64
    INFO 12-10 15:42:47 forward_context.py:58] Batchsize distribution (batchsize, count): [(256, 769), (232, 215), ...]

    $ # 3. The most common batch sizes are 256 and 232, so we can compile the model for these two batch sizes
    $ python3 benchmarks/benchmark_throughput.py --input-len 256 --output-len 256 --model meta-llama/Llama-3.2-1B --load-format dummy --num-scheduler-steps 64 -O "{'level': 3, 'candidate_compile_sizes': [232, 256]}"
    init engine (profile, create kv cache, warmup model) took 80.27 seconds
    Throughput: 15.38 requests/s, 7873.07 total tokens/s, 3936.54 output tokens/s

[youkaichao]

Should we document how to enable/disable custom inductor passes?

not right now until we have some passes with significant perf gain.

[DarkLight1337]

Can you update the Compatibility Matrix with this feature as well?

[youkaichao]

Can you update the Compatibility Matrix with this feature as well?

it's quite complicated, especially for vision language models. Maybe I can just list all the models not supporting torch.compile in the future. The criterion is simple:

• For a text-only model, if the modeling file has support_torch_compile, it means torch.compile is supported. If the super class supports it, then subclass also supports it, e.g. class AriaMoELMModel(LlamaModel): .
• For a multi-modality model, if the text part supports torch.compile, then it is supported as well.

I think mainly cross-attention models do not support torch.compile right now.

[youkaichao]

@DarkLight1337 I checked all the models, and these should be the list of unsupported models. I will also update the add models page to show how to add support for new models shortly.

[bnellnm]

Can you clarify this statement? I think I know what it is trying to say but I think it is a bit ambiguous.

[youkaichao]

do you have better ideas?

[bnellnm]

Maybe something like "torch.compile works best for models that are GPU bound"?