Static quant support for SmoothQuant

[Xia-Weiwen]

Summary
This PR adds static quant support for SmoothQuant by adding a new Int8StaticActivationInt8WeightConfig configuration. Static quantization will generally have better latency & throughput than dynamic quant as it saves the overhead of runtime qparam selection.
In the implementation:

• Activation is only per-tensor quantized to support dynamic shape of activation.
• The SmoothQuantObserver returns act scale along with the smoothing factor.
• The act scale of each layer is set to Int8StaticActivationInt8WeightConfig for transformation of each linear layer.
  • Note: The Int8StaticActivationInt8WeightConfig is not suitable for general static quantization (although it works), users should use PT2E in that case. It's because the act scale for the config are global instead of per-linear-layer, which is the same as Float8StaticActivationFloat8WeightConfig

Test plan
This PR also updates the test cases for SmoothQuant:

• Support CPU-only environment
• Bug fix: the linear module in the UT is not transformed to SmoothQuantLinear since the linear module itself is the parent module.
• Add outliers to example inputs to simulate the case which SmoothQuant is intended to handle.

pytest -sv test/prototype/test_smoothquant.py

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/3089

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures

As of commit abfce41 with merge base 4013764 ():

NEW FAILURES - The following jobs have failed:

• Run Regression Tests / test (CUDA 2.6, linux.g5.12xlarge.nvidia.gpu, torch==2.6.0, cuda, 12.6) / linux-job (gh)
  test/prototype/test_smoothquant.py::TestSmoothQuant::test_smoothquant_static_act_accuracy_alpha_0_25_device_cuda_bfloat16
• Run Regression Tests / test (CUDA 2.7, linux.g5.12xlarge.nvidia.gpu, torch==2.7.0, cuda, 12.6) / linux-job (gh)
  test/prototype/test_smoothquant.py::TestSmoothQuant::test_smoothquant_static_act_accuracy_alpha_0_25_device_cuda_bfloat16
• Run Regression Tests / test (CUDA 2.8, linux.g5.12xlarge.nvidia.gpu, torch==2.8.0, cuda, 12.6) / linux-job (gh)
  test/prototype/test_smoothquant.py::TestSmoothQuant::test_smoothquant_static_act_accuracy_alpha_0_25_device_cuda_bfloat16
• Run Regression Tests / test-nightly (CUDA Nightly, linux.g5.12xlarge.nvidia.gpu, --pre torch --index-url https://downloa... / linux-job (gh)
  test/prototype/test_smoothquant.py::TestSmoothQuant::test_smoothquant_static_act_accuracy_alpha_0_25_device_cuda_bfloat16

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[jerryzh168]

@Xia-Weiwen I think it's better to wait until the Int8Tensor migration is done

[Xia-Weiwen]

@Xia-Weiwen I think it's better to wait until the Int8Tensor migration is done

Thanks for the info