Gradient accumulation implemention

[King4819]

I want to ask that how to implement gradient accumulation on your work. Since my computing resource is single RTX4090 (24GB), so I'm not able to set batch size to 16, thanks !!!

[duanduanduanyuchen]

Hi, you can use GradientCumulativeOptimizerHook. Just set the dict in the config file like this:

data = dict(samples_per_gpu=1)
optimizer_config = dict(type='GradientCumulativeOptimizerHook', cumulative_iters=2)

The total batch_size will be samples_per_gpu*cumulative_iters*num_gpus.

[King4819]

@duanduanduanyuchen Thanks for your reply !