🐛 [Multi-node Full Fine Tuning Example] Fix file path references and disable fused optimizer

[brianlechthaler]

Context

Fix bugs with the Multi-node Full Fine tuning example a

Changelog

What are the changes made in this PR?

• Disable Fused Optimizer:
  • Tensor Parallelism does not support this feature.
• Fix directory references
  • Use ${checkpoint_dir} instead of /tmp/Llama-3.3-70B-Instruct and ${output_dir} instead of /tmp/torchtune/llama3_3_70B/full

Test plan

Please make sure to do each of the following if applicable to your PR. If you're unsure about any one of these just ask and we will happily help. We also have a contributing page for some guidance on contributing.

• Hardware: 2x H200 SXM nodes (16 total GPU) in Nebius eu-north-1 with NCCL over InfiniBand
  • Nebius Platform ID: gpu-h200-sxm
  • Nebius Preset ID: 8gpu-128vcpu-1600gb
• Slurm: Nebius Soperator v1.19 https://github.com/nebius/nebius-solution-library/releases/tag/soperator-v1.19.0-1
• PyTorch version 2.5.1
• testing sbatch script:

# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.

# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree.

# ---------- SBATCH commands ---------- #
#SBATCH --job-name=torchtune-multi-node
#SBATCH --ntasks=2
#SBATCH --nodes=2
#SBATCH --gpus-per-task=8
#SBATCH --cpus-per-task=96

. venv/bin/activate

# ---------- Set env variables ---------- #
# Grab the IP for head node:
# You may need to set this to the fully qualified domain name of your head node
nodes=( $( scontrol show hostnames $SLURM_JOB_NODELIST ) )
nodes_array=($nodes)
head_node=${nodes_array[0]}
head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address)
echo Node IP: $head_node_ip

# You might need to explicitly set the network interface for distributed backends:
# export NCCL_SOCKET_IFNAME=...
# export GLOO_SOCKET_IFNAME=...

export NCCL_DEBUG=INFO

export TORCH_DIST_INIT_BARRIER=1
export LOGLEVEL=INFO

# ---------- Launch training ---------- #
# You probably want to load in a virtual env w/ conda...
# module load conda
# conda activate torchtune
# ...or venv
# source torchtune/bin/activate

SHARED_FS=/home/brian/model # <-- Replace w/ your filesystem
CHECKPOINT_DIR="$SHARED_FS/Llama-3.3-70B-Instruct"
OUTPUT_DIR="$SHARED_FS/Llama3.3-70B-fft-output"
#rsync --progress --stats -a $CHECKPOINT_DIR /tmp
# Adjust sbatch --ntasks and sbatch --nodes above and --nnodes below to your specific node count
srun tune run --nnodes 2 --nproc_per_node 8 --rdzv_id 101 --rdzv_backend c10d --rdzv_endpoint "$head_node_ip:29500" \
    full_finetune_distributed --config ./ttune_sample.yaml checkpoint_dir=$CHECKPOINT_DIR output_dir=$OUTPUT_DIR

• Saved patched TorchTune config file as ./ttune_sample.yaml
• Saved sbatch script as ttune.slurm
• Run with sbatch ttune.slurm
• Output of torchtune run: https://gist.github.com/brianlechthaler/0c42c28e1d6f761619639abe4cdb2dbb

UX

No changes made to API

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/2575

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

Hi @brianlechthaler!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

[ebsmothers]

Hi @brianlechthaler thanks for the PR! Regarding the change to fused optimizer, can you share the error you're seeing? I changed our TP strategy to be compatible with fused optimizer in #2503. (I haven't tested it on a multinode setup myself yet, but I thought it should generalize.)