Dtype gen pass

[WHoutstanding]

PR Category

other

Description

FP32_SENSITIVE_OPS = {
torch.nn.functional.layer_norm: ({2, 3, 4}, {"weight", "bias", "eps"}),
torch.nn.functional.group_norm: ({2, 3, 4}, {"weight", "bias", "eps"}),
torch.nn.functional.batch_norm: ({1, 2, 3, 4}, {"running_mean", "running_var", "weight", "bias"}),
torch.nn.functional.embedding: ({0}, {"weight"}),
}

[paddle-bot]

Thanks for your contribution!

[WHoutstanding]

以torch.nn.functional.layer_norm为例子:
保持"weight", "bias", "eps"为fp32，但是error在：

%hidden_states_1 : [num_users=1] = call_function[target=torch.nn.functional.layer_norm](args = (%hidden_states, (1024,), %L_self_modules_vision_encoder_modules_layers_modules_0_modules_layer_norm1_parameters_weight_, %L_self_modules_vision_encoder_modules_layers_modules_0_modules_layer_norm1_parameters_bias_, %item_1), kwargs = {})

其中hidden_states为fp16，后三个参数我通过查看fx.graph其类型均为fp32，这样还是会失败：

    hidden_states_1 = torch.nn.functional.layer_norm(hidden_states, (1024,), L_self_modules_vision_encoder_modules_layers_modules_0_modules_layer_norm1_parameters_weight_, L_self_modules_vision_encoder_modules_layers_modules_0_modules_layer_norm1_parameters_bias_, item_1);  L_self_modules_vision_encoder_modules_layers_modules_0_modules_layer_norm1_parameters_weight_ = L_self_modules_vision_encoder_modules_layers_modules_0_modules_layer_norm1_parameters_bias_ = item_1 = None
  File "/home/aistudio/miniconda3/envs/GraphNet/lib/python3.10/site-packages/torch/nn/functional.py", line 2910, in layer_norm
    return torch.layer_norm(
RuntimeError: expected scalar type BFloat16 but found Float

AI给出的解释是：
当 torch.layer_norm 被调用时，PyTorch 的分发器（Dispatcher）看到第一个输入是 BFloat16，它会尝试调用 BFloat16 版本的内核。然而，当它检查 weight 和 bias 时，发现它们是 Float32。