Group normalization documentation is incorrect

[albertz]

Describe the bug

This is purely about the documentation.

In the documentation about group normalization, it is stated:

Relation to Layer Normalization: If the number of groups is set to 1, then this operation becomes identical to Layer Normalization.

However, that is not true.

Assume an input tensor x of shape [B,T,F] (batch, time, feature-dim) (time could also be H/W instead; feature-dim can also be the channels).

In layer normalization, the mean you calculate is:

mean = reduce_mean(x, axis=-1, keepdims=True)  # shape [B,T,1]

You normalize just over the feature axis.

In group normalization with G=1 (ignore the group shape then), the mean you calculate is:

mean = reduce_mean(x, axis=[1,2], keepdims=True)  # shape [B,1,1]

You normalize over all axes except the batch axis and the newly added group axis (doesn't matter if G=1).

Or do I misunderstand sth? I wonder because the same wrong statement is in the original group-normalization paper.

The figure from the paper (also here) is also misleading:

In this figure, it looks like layer-normalization normalizes over H/W as well. But this is not the case (at least commonly, and also with the default options).
So, this figure is wrong about layer-normalization (it would just normalize over C, not H/W).
But the figure is correct for group-normalization as you have implemented it (it normalizes over all axes except N/G).

I also formulated the question here.

[bhack]

Have you checked the impl?
Try to follow self.group, group_shape, group_reduction_axes with your case at https://github.com/tensorflow/addons/blob/v0.11.2/tensorflow_addons/layers/normalizations.py#L156

[bhack]

For layer normalization check https://github.com/tensorflow/tensorflow/blob/v2.3.0/tensorflow/python/keras/layers/normalization.py#L1203

[albertz]

Yes, I have. Are you saying I'm wrong, or just double checking?
The code for group-normalization is (with added comments, and prefix, for case G=1):

        B, T, F = tf.shape(x)
        reshaped_inputs = tf.reshape(x, [B, T, 1, F])

        group_shape = tf.keras.backend.int_shape(reshaped_inputs)  # [B,T,1,F]
        group_reduction_axes = list(range(1, len(group_shape)))  # [1,2,3]
        axis = -2 if self.axis == -1 else self.axis - 1  # -2
        group_reduction_axes.pop(axis)  # [1,3] == [T,F]

        mean, variance = tf.nn.moments(
            reshaped_inputs, group_reduction_axes, keepdims=True
        )

For layer-normalization, the code is:

      # self.axis == [-1] == [F]
      mean, variance = nn.moments(inputs, self.axis, keep_dims=True)

[bhack]

So in the original comment do you meant moments on [T,F] with G=1 vs moments on [F] with layer-normalizzation?

[albertz]

So in the original comment do you meant moments on [T,F] with G=1 vs moments on [F] with layer-normalizzation?

Yes exactly. That's not the same. The behavior is very different.

But in the documentation (and even in the paper), you/they write:

Relation to Layer Normalization: If the number of groups is set to 1, then this operation becomes identical to Layer Normalization.

Which is wrong, right?

And the figure is also wrong in the same way. In the figure, it looks like layer-normalization would also calculate the moments over H/W, but normally, you would not do that, and also not with the default options (axis=-1 for LN).

[bhack]

I think it is ok that LN is over the feature map.
About GroupNorm impl. /cc @Smokrow

[bhack]

@albertz Probably this is what you was looking for in the specific TF layer normalizzation https://github.com/tensorflow/tensorflow/blob/v2.3.0/tensorflow/python/keras/layers/normalization.py#L1029-L1036

[bhack]

If you want to open a small PR to better notify this in the documentation you are welcome

[albertz]

I think the GroupNorm implementation is fine. At least it seems to be what is commonly implemented.

I think the LayerNorm implementation is also fine. This seems to be the standard (to just normalize over the feature-dim).

So first of all, I just wanted to confirm that I understood it correct, because it was not totally clear to me. As you seem to confirm this, then I think we just should fix the documentation to be more clear, that GroupNorm with G=1 is not the same as LayerNorm. I can create a PR for that.

@albertz Probably this is what you was looking for in the specific TF layer normalizzation https://github.com/tensorflow/tensorflow/blob/v2.3.0/tensorflow/python/keras/layers/normalization.py#L1029-L1036

This is yet another issue/aspect. Your linked comment in LN is about gamma and beta.
But I was so far only speaking about the statistics, i.e. what exactly is normalized, not about gamma/beta.

(Off topic: @Smokrow Hi from Aachen as well. ☺)

[bhack]

No it is not about only the numerically reproducibility for the gamma and beta but it is also the first part of the claim:

with group size of 1 corresponds to a Layer Normalization that normalizes across height, width, and channal

So they are not talking about implicit default params equivalence. E.g. see the 2nd example in https://keras.io/api/layers/normalization_layers/layer_normalization/

[albertz]

Yes, right. If you use LN and configure it to normalize over all axes (except batch), then gamma/beta is wrong.
Which is not even possible for my example, as T is usually dynamic, not static. Also it doesn't work if height/width is dynamic.

But these are two aspects:

• What axes are normalized over? (The axis argument of tf.nn.moments.)
• What is the shape for the params (gamma/beta)?

LN by default normalizes over the feature-axis, although that can be configured (in TF at least).
LN (at least the TF implementation) seems to imply that the param shape consists exactly of the axes we normalize over. I'm not sure if this in general the standard behavior, or a TF specific decision. Also, this never works if there is any other dynamic axis.

GN exactly defines what axes to normalize over (all except batch and G).
GN with G=1 implies that the param shape is [F].

[bhack]

@seanpmorgan Let me know if you see a margin for a documentation PR or if we could close this.
I don't think that the codowner is available.