[QUESTION] Why is the initialization of the router and experts different in the MoE part?

[mxymxy77]

Why is the initialization of the router and experts different in the MoE part?
The weight parameters of the router are initialized in FP32, while the expert weights are initialized in param_dtype (assuming mixed precision training, the expert weights are initialized in BF16).
Reference link:

1. router init:https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/transformer/moe/router.py#L43
2. experts init:https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/transformer/moe/experts.py#L145

[lk137095576]

same question