You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The number of frames on each rank is not equal, in my run, the number of frames on rank0 is less than total_frames / world_size, so the progress bar will be less than the actual total number of frames. For example, I used 3 gpu, and total 21007 frames(got10k val), and here is some out info. the final progress only 5565*3=16695.
During distributed training, sot model use multi_gpu_test() func to test all videos on several GPUs. But the progress bar is updated on rank 0.
https://github.com/open-mmlab/mmtracking/blob/e79491ec8f0b8c86fda947fbaaa824c66ab2a991/mmtrack/apis/test.py#L160C34-L160C34
The number of frames on each rank is not equal, in my run, the number of frames on rank0 is less than total_frames / world_size, so the progress bar will be less than the actual total number of frames. For example, I used 3 gpu, and total 21007 frames(got10k val), and here is some out info. the final progress only 5565*3=16695.
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ] 16695/21007, 99.7 task/s, elapsed: 167s, ETA: 43s
rank 0 world_size 3 cnt 5565.
rank 1 world_size 3 cnt 7072.
rank 2 world_size 3 cnt 8370.
The text was updated successfully, but these errors were encountered: