feat: add some torch.distributed examples (#315)

* feat: add test_dist_all script

* feat: init mp worker & ray worker

* feat: add test_dist_all script

* feat: add test all_to_all_single uneven

* feat: add test all_to_all_single uneven

* feat: add test all_to_all_single uneven

* feat: add ray all_to_all_single uneven example

* feat: add ray all_to_all_single uneven example

* feat: add ray all_to_all_single uneven example

* feat: add ray all_to_all_single uneven example

* feat: add ray all_to_all_single uneven example

* feat: add ray all_to_all_single uneven example

Author: DefTruth

CONTRIBUTE.md

@@ -11,6 +11,8 @@ Before submitting code, configure pre-commit, for example:
 # fork xlite-dev/LeetCUDA to your own github page, then:
 git clone git@github.com:your-github-page/your-fork-LeetCUDA.git
 cd your-fork-LeetCUDA && git checkout -b test
+# update submodule
+git submodule update --init --recursive --force
 # install pre-commit
 pip3 install pre-commit
 pre-commit install

others/pytorch/distributed/test_all_to_all_single_ray.py

@@ -119,6 +119,7 @@ def run(self) -> torch.Tensor:
 
 
 if __name__ == "__main__":
+    # export RAY_DEDUP_LOGS=0
     world_size = torch.cuda.device_count()
     print(f"world_size: {world_size}")
     if not ray.is_initialized():