Missing pretrain_gpt.py in pytorch distribute pytests

[lhywanggithub]

I could not find the pretrain_gpt.py under tests/pytorch/distributed, which is needed in

TransformerEngine/tests/pytorch/distributed/run_megatron_lm_gpt.sh

Line 120 in ea43b18

NVTE_TORCH_COMPILE=0 NVTE_ALLOW_NONDETERMINISTIC_ALGO=0 NVTE_FLASH_ATTN=1 NVTE_FWD_LAYERNORM_SM_MARGIN=0 NVTE_BWD_LAYERNORM_SM_MARGIN=0 CUDA_DEVICE_MAX_CONNECTIONS=1 NVTE_BIAS_GELU_NVFUSION=0 NVTE_BIAS_DROPOUT_FUSION=0 python -m torch.distributed.launch --use_env --nnodes=1 --nproc_per_node=${NUM_GPUS} ${DIR}/pretrain_gpt.py ${options} 2>&1 | tee $FILENAME

[ptrendx]

@ksivaman

[ksivaman]

@lhywanggithub The distributed tests run megatron-LM. The corresponding files needed, including pretrain_gpt.py can be found there as well. You can follow the commands in the TE test script to get run the tests.