One question about batch norm code when setting use_global_stats True.

[qingqing01]

In the cuDNN impl of batch norm, the code in src/operator/nn/cudnn/cudnn_batch_norm-inl.h is:

      CUDNN_CALL(cudnnBatchNormalizationBackward(
        s->dnn_handle_,
        mode,
        &a,
        &b,
        &a,
        req[cudnnbatchnorm::kGamma] == kWriteTo ? &b: &b_add,
        io_desc_,
        x.dptr_,
        io_desc_,
        dy.dptr_,
        io_desc_,
        dx.dptr_,
        mean_desc_,
        gamma.dptr_,
        dgamma.dptr_,
        dbeta.dptr_,
        param_.eps,
        global_stats ? nullptr : save_mean.dptr_,
global_stats ? nullptr : save_inv_var.dptr_));

https://github.com/apache/incubator-mxnet/blob/master/src/operator/nn/cudnn/cudnn_batch_norm-inl.h#L211

In the cuDNN impl of batch norm, why the savedMean and savedInvVariance are nullptr when global_stats is True? From the cuDNN doc:

savedMean, savedInvVariance
Inputs. Optional cache parameters containing saved intermediate results computed during the forward pass. For this to work correctly, the layer's x and bnScale, bnBias data has to remain unchanged until the backward function is called. Note that both of these parameters can be NULL but only at the same time. It is recommended to use this cache since the memory overhead is relatively small.

https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#cudnnBatchNormalizationBackward

it seems that when these two arguments are nullptr, the cudnnBatchNormalizationBackward will calculate mean/var by using the input X inner this API. I think it is not equivalent to the real backward (see following format ) when global_stats is True and using global mean/variance in forward.

When using global mean/variance,

the forward is:

y = scale * ( (x-global_mean) / sqrt(global_var) ) + bias

the dx is:

dx = scale * dy / sqrt(global_var)

So, I'm a little confused.

[vdantu]

@mxnet-label-bot add [Question]