NeMo 2 Performance instructions

Author: pbelevich

3.test_cases/megatron/nemo/PERFORMANCE.md

@@ -0,0 +1,65 @@
+# Performance
+
+This document describes the process of performance measurements of NeMo 2.x framework.
+
+* [NVIDIA NeMo Performance Summary](https://docs.nvidia.com/nemo-framework/user-guide/latest/performance/performance-summary.html)
+* [NVIDIA NeMo Performance Scripts](https://github.com/NVIDIA/NeMo/tree/main/scripts/performance/llm)
+* [NVIDIA NeMo Compatibility Matrix](https://docs.nvidia.com/nemo-framework/user-guide/latest/softwarecomponentversions.html)
+
+### Create Conda Environment
+
+```bash
+conda create -yn nemo python=3.12
+conda activate nemo
+```
+
+### Install NeMo
+
+Make sure that Nemo version is compatible with the docker image according to the [compatibility matrix](https://docs.nvidia.com/nemo-framework/user-guide/latest/softwarecomponentversions.html)
+
+```bash
+git clone [email protected]:NVIDIA/NeMo.git
+cd NeMo
+git checkout v2.5.0rc0
+pip install -e '.[all]'
+```
+
+Optionally specify where to store the performance results:
+
+```bash
+export NEMORUN_HOME=/fsxl/.../nemo_run
+```
+
+### Build Docker Image
+
+The docker file is supposed to start with `FROM nvcr.io/nvidia/nemo:YY.MM` and continue with EFA installation. Make sure that the docker image is compatible with the Nemo version according to the [compatibility matrix](https://docs.nvidia.com/nemo-framework/user-guide/latest/softwarecomponentversions.html)
+
+```bash
+docker build --progress=plain -t aws-nemo:latest -f Dockerfile .
+enroot import -o ~/aws-nemo.sqsh dockerd://aws-nemo:latest
+```
+
+### Run Performance Test
+
+To enable EFA just export environment variables:
+
+```bash
+export FI_PROVIDER=efa
+export NCCL_DEBUG=INFO
+```
+
+## Recommended model configs
+
+## NVIDIA H100(also applicable to H200)
+
+`NeMo/scripts/performance/recommended_model_configs/model_configs_h100.csv`
+
+| Model     | #-GPUs | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | GA |
+|-----------|--------|-----|-----|-----------------|----|----|----|----|----|----|
+| LLAMA3-8B | 8      | 128 | 1   | 8192            | 1  | 1  | 2  | 1  | 1  | 32 |
+
+```bash
+python -m scripts.performance.llm.pretrain_llama3_8b \
+    --account $(whoami) --partition p5en -i ./aws-nemo.sqsh 
+    --gpu h100 --num_gpus 8 -gb 128 -mb 1 -tp 1 -pp 1 -cp 2 -vp 1 -ep 1
+```