verification of fp8 precision

[kraza8]

Hi, I am using this code for HF transformer model inference. Is there is way I can actually verify that the model is using fp8 precision like print out debug logs or something showing fp8 precision?

fp8_recipe = recipe.DelayedScaling(fp8_format=recipe.Format.E4M3)
def generate_fp8(prompt):
with te.fp8_autocast(enabled=True, fp8_recipe=fp8_recipe):
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
print(f"Input tensor dtype: {inputs['input_ids'].dtype}") # this shows dtype: torch.int64
outputs = model.generate(**inputs, max_length=50)

return tokenizer.decode(outputs[0], skip_special_tokens=True)

thanks

[Advait251206]

The input_ids remaining as torch.int64 is expected and does not indicate whether FP8 is being used. Token IDs are integer indices that are consumed by the embedding layer, so they are never converted to FP8.

To verify that Transformer Engine is actually using FP8, you can try the following:

1. Check the model modules

   fp8_autocast only affects operations implemented by Transformer Engine (e.g., te.Linear, te.LayerNormLinear, etc.). If your Hugging Face model consists entirely of standard torch.nn.Linear layers, simply wrapping the forward pass in fp8_autocast will not automatically convert those layers to FP8.

   You can inspect the model with:

   for name, module in model.named_modules():
       print(name, type(module))

   If you only see standard PyTorch modules (e.g., torch.nn.Linear), then FP8 kernels are likely not being used.

2. Inspect parameter dtypes

   The model parameters themselves typically remain in higher precision (BF16 or FP16). FP8 is generally used for the GEMM computations and temporary tensors, so seeing:

   for name, param in model.named_parameters():
       print(name, param.dtype)

   report torch.bfloat16 or torch.float16 is expected.

3. Use a profiler

   The most reliable way to verify FP8 execution is with Nsight Systems or Nsight Compute. These tools can show whether FP8 Tensor Core kernels are being launched during inference.

4. Enable Transformer Engine logging (if available)

   Depending on the Transformer Engine version, there may be debug or verbose logging options. If you're using a recent release, check the documentation or release notes to see whether FP8-related logging has been added.

If you're using a standard Hugging Face Transformer model without replacing its layers with Transformer Engine modules (or using an integration that performs this replacement), then fp8_autocast alone is generally not sufficient to enable FP8 execution. In that case, the model will continue to execute using its original PyTorch layers even though it is inside the FP8 autocast context.

Could you also share:

• The Hugging Face model you're using (e.g., Llama, Qwen, Mistral, etc.)
• The Transformer Engine version
• The PyTorch version
• Whether you've converted any layers to Transformer Engine modules or are only wrapping inference with fp8_autocast

That would help determine whether FP8 kernels are actually expected to be used in your setup.

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206