[QUESTION] Why BF16 has FP32 for param gradient at the first time unlike FP16 has FP16 for param gradient at the first time and switches them to FP32 when update param?

[renyinCheng001]

Hi, All

I saw that Megatron-LM supports the configuration of FP16 and BF16 mixed-precision data types, and I found that these two different data type configurations correspond to different parameter types and gradient types:

	FP16 Training	BF16 Training
Weight	FP16	BF16
Gradient	FP16	FP32

• When training with FP16, a copy of the FP32 gradient is required before updating the parameters
• When training with BF16, no additional copies are required because the gradient itself is FP32

So, why BF16 has FP32 for param gradient at the first time unlike FP16 has FP16 for param gradient at the first time and switches them to FP32 when update param?

Anyone's reply will be helpful to me, thanks