[Bug] RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one.

[342600694]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• The bug has not been fixed in the latest version(https://github.com/open-mmlab/mmpose).

Environment

Package Version Editable project location

---

actionlib 1.14.0
addict 2.4.0
aliyun-python-sdk-core 2.15.2
aliyun-python-sdk-kms 2.16.4
angles 1.9.13
attrs 24.2.0
auditwheel 6.0.0
bondpy 1.8.6
camera-calibration 1.17.0
camera-calibration-parsers 1.12.0
catkin 0.8.10
certifi 2024.7.4
cffi 1.17.0
charset-normalizer 3.3.2
chumpy 0.70
click 8.1.7
cmake 3.30.2
colorama 0.4.6
contourpy 1.1.1
controller-manager 0.19.6
controller-manager-msgs 0.19.6
coverage 7.6.1
crcmod 1.7
cryptography 43.0.0
cv-bridge 1.16.2
cycler 0.12.1
Cython 3.0.11
diagnostic-analysis 1.11.0
diagnostic-common-diagnostics 1.11.0
diagnostic-updater 1.11.0
diff-gaussian-rasterization 0.0.0
dynamic-reconfigure 1.7.3
exceptiongroup 1.2.2
filelock 3.14.0
flake8 7.1.1
fonttools 4.53.1
gazebo_plugins 2.9.2
gazebo_ros 2.9.2
gencpp 0.7.0
geneus 3.0.0
genlisp 0.4.18
genmsg 0.6.0
gennodejs 2.0.2
genpy 0.6.15
idna 3.8
image-geometry 1.16.2
importlib_metadata 8.4.0
importlib_resources 6.4.4
iniconfig 2.0.0
interactive-markers 1.12.0
interrogate 1.7.0
isort 4.3.21
Jinja2 3.1.4
jmespath 0.10.0
joint-state-publisher 1.15.1
joint-state-publisher-gui 1.15.1
json-tricks 3.17.3
kiwisolver 1.4.5
laser_geometry 1.6.7
lit 18.1.8
Markdown 3.7
markdown-it-py 3.0.0
MarkupSafe 2.1.5
matplotlib 3.7.5
mccabe 0.7.0
mdurl 0.1.2
message-filters 1.16.0
mmcv 2.1.0
mmdet 3.2.0
mmengine 0.10.4
mmpose 1.3.2 /home/ubuntu/LHD/mmpose
model-index 0.1.11
mpmath 1.3.0
munkres 1.1.4
networkx 3.1
numpy 1.24.4
opencv-python 4.10.0.84
opendatalab 0.0.10
openmim 0.3.9
openxlab 0.1.1
ordered-set 4.1.0
oss2 2.17.0
packaging 24.1
pandas 2.0.3
parameterized 0.9.0
pillow 10.4.0
pip 24.2
platformdirs 4.2.2
pluggy 1.5.0
py 1.11.0
pycocotools 2.0.7
pycodestyle 2.12.1
pycparser 2.22
pycryptodome 3.20.0
pyelftools 0.31
pyflakes 3.2.0
Pygments 2.18.0
pyparsing 3.1.4
pytest 8.3.2
pytest-runner 6.0.1
python-dateutil 2.9.0.post0
python-qt-binding 0.4.4
pytz 2023.4
PyYAML 6.0.2
qt-dotgraph 0.4.2
qt-gui 0.4.2
qt-gui-cpp 0.4.2
qt-gui-py-common 0.4.2
requests 2.28.2
resource_retriever 1.12.7
rich 13.4.2
ros_numpy 0.0.5
rosbag 1.16.0
rosboost-cfg 1.15.8
rosclean 1.15.8
roscreate 1.15.8
rosgraph 1.16.0
roslaunch 1.16.0
roslib 1.15.8
roslint 0.12.0
roslz4 1.16.0
rosmake 1.15.8
rosmaster 1.16.0
rosmsg 1.16.0
rosnode 1.16.0
rosparam 1.16.0
rospy 1.16.0
rosservice 1.16.0
rostest 1.16.0
rostopic 1.16.0
rosunit 1.15.8
roswtf 1.16.0
rqt_action 0.4.9
rqt_bag 0.5.1
rqt_bag_plugins 0.5.1
rqt_console 0.4.11
rqt_dep 0.4.12
rqt_graph 0.4.14
rqt_gui 0.5.3
rqt_gui_py 0.5.3
rqt-image-view 0.4.17
rqt_launch 0.4.9
rqt_logger_level 0.4.11
rqt-moveit 0.5.10
rqt_msg 0.4.10
rqt_nav_view 0.5.7
rqt_plot 0.4.13
rqt_pose_view 0.5.11
rqt_publisher 0.4.10
rqt_py_common 0.5.3
rqt_py_console 0.4.10
rqt-reconfigure 0.5.5
rqt-robot-dashboard 0.5.8
rqt-robot-monitor 0.5.14
rqt_robot_steering 0.5.12
rqt_runtime_monitor 0.5.9
rqt-rviz 0.7.0
rqt_service_caller 0.4.10
rqt_shell 0.4.11
rqt_srv 0.4.9
rqt_tf_tree 0.6.3
rqt_top 0.4.10
rqt_topic 0.4.13
rqt_web 0.4.10
rviz 1.14.20
scipy 1.10.1
sensor-msgs 1.13.1
setuptools 60.2.0
shapely 2.0.6
six 1.16.0
smach 2.5.1
smach-ros 2.5.1
smclib 1.8.6
sympy 1.13.2
tabulate 0.9.0
termcolor 2.4.0
terminaltables 3.1.10
tf 1.13.2
tf-conversions 1.13.2
tf2-geometry-msgs 0.7.6
tf2-kdl 0.7.6
tf2-py 0.7.6
tf2-ros 0.7.6
tomli 2.0.1
topic-tools 1.16.0
torch 2.0.0+cu118
torchaudio 2.0.0+cu118
torchvision 0.15.0+cu118
tqdm 4.65.2
triton 2.0.0
typing_extensions 4.12.2
tzdata 2024.1
urllib3 1.26.19
wheel 0.44.0
xacro 1.14.15
xdoctest 1.2.0
xtcocotools 1.14.3
yapf 0.40.2
zipp 3.20.1

Reproduces the problem - code sample

I did not make any modifications to the training code, I used the training code provided in mmpose
def main():
args = parse_args()

# load config
cfg = Config.fromfile(args.config)

# merge CLI arguments to config
cfg = merge_args(cfg, args)

# set preprocess configs to model
#如果 cfg.model 中没有 data_preprocessor，那么将 preprocess_cfg 的配置作为 data_preprocessor 的值进行设置。
#如果 preprocess_cfg 在配置中存在，则将其传递给模型的 data_preprocessor。
if 'preprocess_cfg' in cfg:
    cfg.model.setdefault('data_preprocessor',
                         cfg.get('preprocess_cfg', {}))

# build the runner from config
runner = Runner.from_cfg(cfg)

# start training
runner.train()

Reproduces the problem - command or script

I made modifications to RTMO. When training on a single graphics card, there was no problem running train.exe. However, when distributing, an error occurred during training.

Reproduces the problem - error message

untimeError: RuntimeErrorExpected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel, and by
making sure all forward function outputs participate in calculating loss.
If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's forward function. Please include the loss function and the structure of the return value of forward of your module when reporting this issue (e.g. list, dict, iterable).
Parameter indices which did not receive grad for rank 2: 362 363 364 365 366 367 368 369 394 395 396 397 398 399 400 401
In addition, you can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print out information about which particular parameters did not receive gradient on this rank as part of this error: RuntimeError
Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel, and by
making sure all forward function outputs participate in calculating loss.
If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's forward function. Please include the loss function and the structure of the return value of forward of your module when reporting this issue (e.g. list, dict, iterable).
Parameter indices which did not receive grad for rank 3: 362 363 364 365 366 367 368 369 394 395 396 397 398 399 400 401
In addition, you can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print out information about which particular parameters did not receive gradient on this rank as part of this error:
Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel, and by
making sure all forward function outputs participate in calculating loss.
If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's forward function. Please include the loss function and the structure of the return value of forward of your module when reporting this issue (e.g. list, dict, iterable).
Parameter indices which did not receive grad for rank 1: 362 363 364 365 366 367 368 369 394 395 396 397 398 399 400 401
In addition, you can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print out information about which particular parameters did not receive gradient on this rank as part of this error
model = self.train_loop.run() # type: ignore
File "/home/ubuntu/anaconda3/envs/mmpose/lib/python3.8/site-packages/mmengine/runner/loops.py", line 96, in run
self.run_epoch()
File "/home/ubuntu/anaconda3/envs/mmpose/lib/python3.8/site-packages/mmengine/runner/loops.py", line 113, in run_epoch
self.run_iter(idx, data_batch)
File "/home/ubuntu/anaconda3/envs/mmpose/lib/python3.8/site-packages/mmengine/runner/loops.py", line 129, in run_iter
outputs = self.runner.model.train_step(
File "/home/ubuntu/anaconda3/envs/mmpose/lib/python3.8/site-packages/mmengine/model/wrappers/distributed.py", line 121, in train_step
losses = self._run_forward(data, mode='loss')
File "/home/ubuntu/anaconda3/envs/mmpose/lib/python3.8/site-packages/mmengine/model/wrappers/distributed.py", line 161, in _run_forward
results = self(**data, mode=mode)
File "/home/ubuntu/anaconda3/envs/mmpose/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/mmpose/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1139, in forward
if torch.is_grad_enabled() and self.reducer._rebuild_buckets():
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel, and by
making sure all forward function outputs participate in calculating loss.
If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's forward function. Please include the loss function and the structure of the return value of forward of your module when reporting this issue (e.g. list, dict, iterable).
Parameter indices which did not receive grad for rank 0: 362 363 364 365 366 367 368 369 394 395 396 397 398 399 400 401
In addition, you can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print out information about which particular parameters did not receive gradient on this rank as part of this error

Additional information

No response