Multi-GPU DistributedDataParallel Mode

[ameicler]

Hi,

Thanks for your work.

How do you get to train on multi-gpus using the recommended way of the repo?

Multi-GPU DistributedDataParallel Mode (✅ recommended)
You will have to pass python -m torch.distributed.run --nproc_per_node, followed by the usual arguments.

$ python -m torch.distributed.run --nproc_per_node 2 train.py --batch 64 --data coco.yaml --weights yolov5s.pt --device 0,1

I've passed "0,1,2,3" as device argument but this seems to do only Multi-GPU DataParallel Mode (not recommended).

Cheers

[ameicler]

Here are the logs I'm seeing, the 4 GPUs are indeed visible but it seems to only look at one when allocating memory

YOLOv5 🚀 torch 1.11.0+cu102 CUDA:0 (Tesla K80, 11441MiB)
                             CUDA:1 (Tesla K80, 11441MiB)
                             CUDA:2 (Tesla K80, 11441MiB)
                             CUDA:3 (Tesla K80, 11441MiB)
...
Overriding model.yaml nc=80 with nc=4
Overriding model.yaml anchors with anchors=2.1467

                 from  n    params  module                                  arguments                     
  0                -1  1      7040  yolov5.models.common.Conv               [3, 64, 6, 2, 2]              
  1                -1  1     73984  yolov5.models.common.Conv               [64, 128, 3, 2]               
  2                -1  3    156928  yolov5.models.common.C3                 [128, 128, 3]                 
  3                -1  1    295424  yolov5.models.common.Conv               [128, 256, 3, 2]              
  4                -1  6   1118208  yolov5.models.common.C3                 [256, 256, 6]                 
  5                -1  1   1180672  yolov5.models.common.Conv               [256, 512, 3, 2]              
  6                -1  9   6433792  yolov5.models.common.C3                 [512, 512, 9]                 
  7                -1  1   3540480  yolov5.models.common.Conv               [512, 768, 3, 2]              
  8                -1  3   5611008  yolov5.models.common.C3                 [768, 768, 3]                 
  9                -1  1   7079936  yolov5.models.common.Conv               [768, 1024, 3, 2]             
 10                -1  3   9971712  yolov5.models.common.C3                 [1024, 1024, 3]               
 11                -1  1   2624512  yolov5.models.common.SPPF               [1024, 1024, 5]               
 12                -1  1    787968  yolov5.models.common.Conv               [1024, 768, 1, 1]             
 13                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 14           [-1, 8]  1         0  yolov5.models.common.Concat             [1]                           
 15                -1  3   6200832  yolov5.models.common.C3                 [1536, 768, 3, False]         
 16                -1  1    394240  yolov5.models.common.Conv               [768, 512, 1, 1]              
 17                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 18           [-1, 6]  1         0  yolov5.models.common.Concat             [1]                           
 19                -1  3   2757632  yolov5.models.common.C3                 [1024, 512, 3, False]         
 20                -1  1    131584  yolov5.models.common.Conv               [512, 256, 1, 1]              
 21                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 22           [-1, 4]  1         0  yolov5.models.common.Concat             [1]                           
 23                -1  3    690688  yolov5.models.common.C3                 [512, 256, 3, False]          
 24                -1  1    590336  yolov5.models.common.Conv               [256, 256, 3, 2]              
 25          [-1, 20]  1         0  yolov5.models.common.Concat             [1]                           
 26                -1  3   2495488  yolov5.models.common.C3                 [512, 512, 3, False]          
 27                -1  1   2360320  yolov5.models.common.Conv               [512, 512, 3, 2]              
 28          [-1, 16]  1         0  yolov5.models.common.Concat             [1]                           
 29                -1  3   5807616  yolov5.models.common.C3                 [1024, 768, 3, False]         
 30                -1  1   5309952  yolov5.models.common.Conv               [768, 768, 3, 2]              
 31          [-1, 12]  1         0  yolov5.models.common.Concat             [1]                           
 32                -1  3  10496000  yolov5.models.common.C3                 [1536, 1024, 3, False]        
 33  [23, 26, 29, 32]  1     46152  yolov5.models.yolo.Detect               [4, [[0, 1, 2, 3], [0, 1, 2, 3], [0, 1, 2, 3], [0, 1, 2, 3]], [256, 512, 768, 1024]]
Model summary: 607 layers, 76162504 parameters, 76162504 gradients, 110.2 GFLOPs

Transferred 786/795 items from /tmp/tmpkxt8_b1p/yolov5l6.pt
�[34m�[1mAutoBatch: �[0mComputing optimal batch size for --imgsz 1280
�[34m�[1mAutoBatch: �[0mCUDA:0 (Tesla K80) 11.17G total, 0.60G reserved, 0.58G allocated, 10.00G free
      Params      GFLOPs  GPU_mem (GB)  forward (ms) backward (ms)                   input                  output
    76162504       440.7         2.861         361.8         631.5      (1, 3, 1280, 1280)                    list
    76162504       881.4         5.083         644.2          1005      (2, 3, 1280, 1280)                    list
    76162504        1763         9.240          1216          1923      (4, 3, 1280, 1280)                    list
CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 11.17 GiB total capacity; 10.69 GiB already allocated; 13.25 MiB free; 10.74 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
CUDA out of memory. Tried to allocate 400.00 MiB (GPU 0; 11.17 GiB total capacity; 10.30 GiB already allocated; 379.25 MiB free; 10.38 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
�[34m�[1mAutoBatch: �[0mUsing batch-size 3 for CUDA:0 10.06G/11.17G (90%)
Scaled weight_decay = 0.00025593750000000003
�[34m�[1moptimizer:�[0m SGD with parameter groups 131 weight (no decay), 135 weight, 135 bias
WARNING: DP not recommended, use torch.distributed.run for best DDP Multi-GPU results.
See Multi-GPU Tutorial at https://github.com/ultralytics/yolov5/issues/475 to get started.