Slow performance on ROCm system #18
Description
Hi there,
I attempted to run the benchmark at problem scale 8 on a ROCm system with AMD GPUs. However, the training is taking an extremely long time (approximately 1 hour per epoch), and I am seeing several warning messages such as:
...
GridwiseOp: Problemsize descriptor dimension check failure
Warning: Workspace for DeviceGroupedConvBwdWeightTwoStage_Xdl_CShuffle::Argument is not allocated, use SetWorkSpacePointer.
...
/scratch/pyenvs/scaffold/lib64/python3.11/site-packages/torch/autograd/graph.py:829: UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout\
contract, or that the param's strides changed since DDP was constructed. This is not an error, but may impair performance.
grad.sizes() = [6, 64, 1, 1, 1], strides() = [64, 1, 1, 1, 1]
bucket_view.sizes() = [6, 64, 1, 1, 1], strides() = [64, 1, 64, 64, 64] (Triggered internally at /pytorch/torch/csrc/distributed/c10d/reducer.cpp:334.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
[06:13:20][trainer.py:405][rank=0][INFO] Done warmup. Took 1640s
I did not observe these warnings when running on a CUDA system. I am wondering:
1. Could these warnings be contributing to the significantly longer runtime?
2. Are there recommended ways to mitigate or eliminate them?
For reference, my benchmark configuration is shown below:
dataset_dir: "/scratch/scaffold_benchmark/datasets_ps8"
checkpoint_interval: 55
fract_base_dir: "fractals_ps8"
n_categories: 5
n_instances_used_per_fractal: 145
problem_scale: 8
unet_bottleneck_dim: 3
seed: 42
batch_size: 1
optimizer: "ADAM"
num_shards: 2
shard_dim: 2
variance_threshold: 0.17
n_fracts_per_vol: 3
val_split: 25
epochs: 50
learning_rate: .0001
disable_scheduler: 1
more_determinism: 0
datagen_from_scratch: 0
train_from_scratch: 1
dist: 1
torch_amp: 1
framework: "torch"
checkpoint_dir: "checkpoints"
loss_freq: 1
normalize: 1
warmup_epochs: 1
dataset_reuse_enforce_commit_id: 0
Thank you for your attention.