Qualcomm AI Engine Direct - [GenAI Pipeline] Multimodal LLM & LLM Preparation and Quantization

[DannyYuyang-quic]

Summary

First PR of Phase 2 Stream A. This PR makes the GenAI Pipeline runnable for
LLM and MLLM models through the front half of the flow:

CLI -> model lookup -> source transforms -> model preparation
    -> dataset-backed calibration -> quantization -> save QDQ model

examples/ and the legacy llama.py path remain unchanged. Compilation and
device inference remain in Stream B / PR-A2.

What's Included

CLI Entry Point: cli.py

Adds a GenAI Pipeline CLI that resolves model, dataset, and quantization options, builds PipelineContext, and invokes model preparation and quantization. FP16 omits quantization; QAT, embedding quantization, and attention sink are rejected for now.
The rejected feature will be raised in following PR.

Registry-Backed Model lookup: model_lookup.py

Maps a user-facing model name to the model-specific inputs consumed by the pipeline. Callers resolve one registry config, then derive from that same entry:

• Model construction: component and graph keyed constructors, including
  decoder decode/quantize graphs, optional prefill graphs, and multimodal
  embedding variants.
• Source preparation: component-keyed state-dict weight transforms and
  module G2G transforms, plus the Hugging Face state-dict loader when needed.
• Pipeline configuration: component sharding, quantization dtypes and
  recipe classes, and LLM or MLLM loader/quantizer adapters.

This centralizes model-family branching at the registry boundary; the model preparation and quantization stages consume only the derived component and graph keyed configuration.

LLM and Multimodal LLM model components: model_components/

Adds pipeline import surfaces for decoder, token embedding, and encoder components over existing examples implementations.

Module G2G Transforms: source_transform/:

Collects existing llama.py transformations into reusable pipeline functions:

• Weight transforms.
• Model graph-to-graph (G2G) transforms.

LLM and MLLM Model Loading and Preparation: strategies/model_preparation/:

ExecuTorchModelPreparationStrategy implements the shared preparation flow:

1. Load one weighted module per component and graph variant.
2. Read per-graph metadata.
3. Build model-native example inputs for each graph.
4. Select one weight-sharing module per component.
5. Apply component-level module transforms.
6. Load the tokenizer.
7. Build the model-specific inference helper used by quantization, if any.
8. Optionally export the tokenizer for runtime.
9. Extract a tokenizer chat template, falling back to extra_options.

LLMLoaderAdapter handles text decoders; MLLMLoaderAdapter handles multimodal models. The default adapter owns shared tokenizer and transforms logic.

Dataset stack: datasets/

Adds typed dataset options, lookup, loaders, calibration adapters, and LLM/MLLM collators. It supports random fallback data, lm-eval samples, JSON messages, Hugging Face chat datasets, and multimodal messages.

generate_calibration_data() now receives example_inputs explicitly because collators require the calibration-graph signature.

Quantization: strategies/quantization/:

ExecuTorchQuantizationStrategy implements native PTQ for component and graph keyed models:

1. Create one QNN quantizer per component graph.
2. Export every graph variant and prepare it with observers.
3. Initialize observers for deploy graphs.
4. Build real calibration inputs through the calibration data adapter.
5. Run PTQ calibration on graphs that consume real data.
6. Convert prepared graphs to QDQ modules.
7. Propagate encodings from quantize graphs to deploy graphs.
8. Remove quantize-only graphs from the output.

Decoder and token embedding use separate quantize and deploy graphs; encoders use one shared graph. Encodings are propagated before lowering.

Quantizer creation belongs to the strategy; LLM/MLLM adapters own only model-family-specific export, preparation, conversion, and calibration.

Quantization Helpers: quant_utilities.py:

Provides QNN quantizer construction, recipe application, QDQ saving, logits/KV-cache attributes, and encoding propagation. KV-cache override is enabled only when n_cache_layers is provided.

Quantization Output

QuantizationOutputConfig now returns component and graph keyed GraphBundle objects. Each bundle carries the quantized graph module, export inputs, metadata, and optional quantized IO dtypes.
Quantize-only graphs are removed after their encodings have been propagated, so downstream compilation sees only deployment graphs.

Tests

Covers CLI/config construction, source transforms, datasets and collators, loader adapters, model preparation, quantization adapters and strategy, and preparation/quantization stage integration.

PR Review Checklist

• All dependencies are injected via constructor with sensible defaults - Yes.
• All external calls are behind injectable interfaces - Yes
• Hugging Face, TokenizerWrapper, torch.export, PT2E, QNN quantizer construction, dataset
  loading, and calibration execution sit behind adapters or lookup-created
  callables.
• Unit tests cover the new public behavior - Yes.
• Docstrings on public classes and methods - Yes.
• Type annotations on function signatures - Yes.
• Logging follows the strategy in the LLD - Yes; info for stage-level progress, debug for per-step details, warning for degraded/normalized CLI behavior. No existing behavior changed - Yes
• examples/ is not modified and llama.py remains the reference path. - Yes

Related PRs

Phase 2 flow. PR-A1 and PR-B1 are independent and can land in either order.
PR-B2 depends on PR-B1. PR-A2 depends on PR-A1 and PR-B2, and closes Phase 2 by wiring the runner and adding the parity test that gates Phase 3 deletion of the legacy flow.

PR-A1 (this PR) ────────────────┐
                                ├─► PR-A2
PR-B1 ───────────► PR-B2 ───────┘

Phase 1 (merged):

• PR 1: Core data model, engine routing & exceptions: Qualcomm AI Engine Direct - [GenAI Pipeline] PR1: Core Data Model, Engine Routing & Exceptions #20409
• PR 2: Strategy interfaces & stage wrappers: Qualcomm AI Engine Direct - [GenAI Pipeline] PR2: Strategy Interfaces & Stage Wrappers #20795
• PR 3: Pipeline orchestrator: Qualcomm AI Engine Direct - [GenAI Pipeline] PR3: Pipeline Orchestrator #21149
• PR 4: Adapter interfaces, default implementations & dataset providers: Qualcomm AI Engine Direct - [GenAI Pipeline] PR4: Adapter interfaces, default implementations & dataset providers #21751
• PR 5: Model preparation & quantization strategy implementations: Qualcomm AI Engine Direct - [GenAI Pipeline] PR5: Model preparation & quantization strategy implementations #21899
• PR 6: Compilation & inference strategy implementations: Qualcomm AI Engine Direct - [GenAI Pipeline] PR6: Compilation & inference strategy implementations #22284
• PR 7: Integration & E2E tests: cancelled, will be raised after Phase 2 is done.

Phase 2:

• PR A1: Model registry, model_lookup, CLI, Multimodal LLM & LLM model preparation,
  dataset-backed calibration, component/graph-aware quantization, and encoding
  reconciliation. [This PR].
• PR B1: Compilation foundation: Qualcomm AI Engine Direct - [GenAI Pipeline Phase 2] PRB1 - Compilation foundation #22846
• PR B2: Multi-graph lowering, weight sharing, sharding, and spill-fill.
  Depends on PR-B1.
• PR A2: Device-runner adapter, pipeline-runner wiring, README, and E2E parity
  test. Depends on PR-A1 and PR-B2.

Test plan

Run only tests added in this PR:

python -m pytest \
  backends/qualcomm/genai_pipeline/tests/strategies/model_preparation/ \
  backends/qualcomm/genai_pipeline/tests/strategies/quantization/ \
  backends/qualcomm/genai_pipeline/tests/source_transform/ \
  -v

Result:

126 passed, 2 subtests passed in 11.59s

Run all genai_pipeline tests:

python -m pytest backends/qualcomm/genai_pipeline/tests/ -v

Result:

238 passed, 13 subtests passed in 14.37s

Run all tests with coverage:

python -m pytest backends/qualcomm/genai_pipeline/tests/ \
  --cov=backends/qualcomm/genai_pipeline \
  --cov-config=backends/qualcomm/.coveragerc \
  --cov-report=term-missing

Result:

238 passed, 13 subtests passed in 17.41s

Confirm the legacy flow is unaffected:

python -c "
from executorch.examples.qualcomm.oss_scripts.llama.llama import _build_parser, export_llama
from executorch.examples.qualcomm.oss_scripts.llama.wrappers import MultiModalManager, HybridAttentionSinkEvictor
print('LEGACY IMPORTS OK')
"

Result:

`LEGACY IMPORTS OK`

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23050

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 14 Awaiting Approval, 1 New Failure

As of commit 4e19eb9 with merge base 9520b05 ():

AWAITING APPROVAL - The following workflows need approval before CI can run:

• Build documentation (gh)
• Build Presets (gh)
• Lint (gh)
• periodic (gh)
• pull (gh)
• Test ARM Backend (gh)
• Test CoreML Backend (gh)
• Test Cortex-M Backend (gh)
• Test NXP Backend (gh)
• Test QNN Backend (gh)
• Test QNN Windows MSVC build (gh)
• Test Vulkan Backend (gh)
• Test WebGPU Backend (gh)
• Test XNNPACK Backend (gh)

NEW FAILURE - The following job has failed:

• Cadence Build & Test / Resolve CI docker image / resolve (gh)
  ##[error]Refusing to check out fork pull request code from a 'pull_request_target' workflow. This workflow runs with the base repository's GITHUB_TOKEN, secrets, default-branch cache scope, and runner access. Fetching and executing a fork's code in that trusted context commonly leads to "pwn request" vulnerabilities. To opt in, review the risks at https://gh.io/securely-using-pull_request_target and set 'allow-unsafe-pr-checkout: true' on the actions/checkout step.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[DannyYuyang-quic]

@pytorchbot label "release notes: qualcomm"

[DannyYuyang-quic]

Hi @psiddh,

TL;DR:

This PR makes the GenAI Pipeline runnable through backends/qualcomm/genai_pipeline/cli.py for the models supported by the existing llama.py flow. It covers model preparation and basic dataset-backed PTQ calibration, producing quantized QDQ models.

The existing examples/ and legacy llama.py flow remain unchanged.
Advanced features, including QAT and AttentionSink, are intentionally out of scope for GenAI Pipeline and
will be introduced in a follow-up PR.

Together with PR B1, Compilation Foundation (#22846), the following commands
produce quantized QDQ models:

LLM

python -m backends.qualcomm.genai_pipeline.cli --model gemma3-1b --soc SM8750 --calib-samples examples/qualcomm/oss_scripts/llama/assets/samples/text.json --compile-only --max-seq-len 1024

Multimodal LLM

Vision-Language model

python -m backends.qualcomm.genai_pipeline.cli --model internvl3_1b --soc SM8750 --calib-samples examples/qualcomm/oss_scripts/llama/assets/samples/vision.json --compile-only --max-seq-len 1024

Audio-Language model

python -m backends.qualcomm.genai_pipeline.cli  --model granite_speech_3_3-2b --soc SM8750 --calib-samples examples/qualcomm/oss_scripts/llama/assets/samples/audio --compile-only --max-seq-len 1024

Please have a look. Thanks!

cc: @shewu-quic @winskuo-quic @qti-horodnic