Cherry-pick fused_adam_cuda related patches and add its unit tests to the ROCm extension test script#93

Open

hubertlu-tw wants to merge 5 commits intomasterfrom dev/hubertlu/fused_adam_cuda

+858-444

Commits on Sep 9, 2022

Add dist_adam test to the extension unit test script
hubertlu-tw
committed
Add distributed_fused_adam_v2 and _v3 back
hubertlu-tw
committed
Store optimizer state_dict in uint8 tensor instead of bytearray (NVIDIA#1446 )

timmoon10
authored and
hubertlu-tw
committed
Optimize distributed Adam kernels and implementation (NVIDIA#1463 )

timmoon10
authored and
hubertlu-tw
committed
Revert "Add distributed_fused_adam_v2 and _v3 back"
hubertlu-tw
committed