Correct for gradient bias in GRPO style reward centering.

[maitchison]

Motivation

GRPO style centering introduces bias into the gradient estimate. The bias introduced is equal to $$\frac{G-1}{G}$$, where $$G$$ is the group size. While this factor is generally small, and can be incorporated into the learning rate, it would be prefered to not have learning rate dependant on group size in this way.

References

• See https://arxiv.org/pdf/2503.20783 (Pg.14)

Changes

• Added a correction to the advantages to adjust for the bias introduced by GRPO reward centering.

[Tiiiger]

hi @maitchison thanks for the contribution! but I don't think this technique is super common these days so we will not have it in the official cookbook. Feel free to add it in your experiments though.

[maitchison]

OK, no worries, thanks for taking a look. I've now closed the PR.