updated mount_fsx.sh to configure FSxL EFA client #849

mayankgupta14 · 2025-09-10T23:13:44Z

Issue #, if available:

Description of changes:

Updated the mount_fsx.sh to install and configure EFA for FSxL on supported instance types.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

KeitaW · 2025-09-11T00:13:37Z

1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/mount_fsx.sh

+    case "$instance_type" in
+        p6-b200.*|p6e-gb200.*|p4d.*|p5.*|p5e.*|p5en.*|trn1.*|trn1n.*|c5n.*|c6gn.*|c6in.*|c7gn.*|g6.*|g6e.*|m5dn.*|m5n.*|m5zn.*|m6a.*|m6i.*|m6in.*|r5dn.*|r5n.*|r6in.*|x2gd.*)


Not all p5.*, g* instances support EFA (ex: p5.4xl, g6.xl)?

Agreed, oversight on my side, thank you. Will update, and in fact will update the function to check for efa device and drivers on the instance instead. Will address the other concerns from Matt as well.

nghtm · 2025-09-11T01:09:40Z

1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/mount_fsx.sh

+        return 0
+    fi
+
+    # EFA-supported instance types


WHile I agree with this methodolog for fetching instance type, just because customer is using EFA supported instance, doesn't mean that customer will automatically have deployed EFA enabled FSxL FileSystem. Both need to be true

So, we are checking the instance type to make sure it's supported by EFA and is supported by EFA enabled FSxL to configure EFA<>FSxL client. There is no drawback to having this client installed. We can't check the FSxL is created with EFA enabled or not unless we make the describe fsx api call.
While I agree, both needs to be true for comms to work over EFA, it's not required for client installation. We need to make sure the instance is supported and EFA drivers are installed on the instance for client configure.

I'll update the function to check for EFA drivers on the instance.

nghtm · 2025-09-11T01:10:16Z

1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/mount_fsx.sh

+        p6-b200.*|p6e-gb200.*|p4d.*|p5.*|p5e.*|p5en.*|trn1.*|trn1n.*|c5n.*|c6gn.*|c6in.*|c7gn.*|g6.*|g6e.*|m5dn.*|m5n.*|m5zn.*|m6a.*|m6i.*|m6in.*|r5dn.*|r5n.*|r6in.*|x2gd.*)
+            echo "[INFO] Instance type $instance_type supports EFA"
+
+            # Download EFA configuration script with validation


What happens if FSxL is not EFA enabled? Is their potential for this install to create issues if customer is using non-EFA enabled FileSystem?

If the customer is using Non-EFA enabled FSxL, the comms fallback to TCP automatically. Installing EFA client on the instance has no drawback.

nghtm · 2025-09-11T01:10:41Z

1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/mount_fsx.sh

    echo "Mount_fsx called with fsx_dns_name: $FSX_DNS_NAME, fsx_mountname: $FSX_MOUNTNAME"
    echo "Using mount_point: $MOUNT_POINT"
    echo "LUSTRE CLIENT CONFIGURATION $(print_lustre_version)"
+    configure_efa_lustre


Same comment as above, what if FSxL FileSystem is not EFA enabled?

If the customer is using Non-EFA enabled FSxL, the comms fallback to TCP automatically. Installing EFA client on the instance has no drawback.

KeitaW · 2025-09-11T05:39:54Z

Also, do we have any usecase Cx can benefit from FSxL + EFA? I have not seen any case so far.

elgalu · 2025-09-12T11:31:39Z

1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/mount_fsx.sh

+            ansible localhost -m ansible.builtin.unarchive -a "src=/tmp/configure-efa-fsx-lustre-client.zip dest=/tmp remote_src=yes"
+
+            # Make script executable and run it
+            ansible localhost -b -m ansible.builtin.file -a "path=/tmp/configure-efa-fsx-lustre-client.sh mode='0755'"


localhost | FAILED! => {
"changed": false,
"msg": "file (/tmp/configure-efa-fsx-lustre-client.sh) is absent, cannot continue",
"path": "/tmp/configure-efa-fsx-lustre-client.sh",
"state": "absent"
}

should be /tmp/configure-efa-fsx-lustre-client/setup.sh

good catch, thank you.

elgalu · 2025-09-12T16:02:06Z

Trying...

mount -t lustre -o noatime,flock,_netdev,x-systemd.automount,x-systemd.requires=network-online.target,net=efa fs-05b6aafaa26ba451a.fsx.eu-north-1.amazonaws.com@tcp:/xduwpbev /mnt/leo2

elgalu · 2025-09-12T20:30:57Z

mount -t lustre -o noatime,flock,_netdev,x-systemd.automount,x-systemd.requires=network-online.target,net=efa fs-05b6aafaa26ba451a.fsx.eu-north-1.amazonaws.com@tcp:/xduwpbev /mnt/leo2

no no, so net=efa doesn't exist, I guess we don't need to specify anything to mount -t lustre ... right?

elgalu · 2025-09-16T14:40:49Z

One tip, you need to disable this when the Lustre AZ is different from the instance AZ otherwise write to /fsx tilts forever and I had to wait 60-90 minutes until it timed out. I added I check on mount-fsx to fix it:

mount-fsx: Cross-AZ detected: EFA mount will be disabled

…not and if the FSx is in the same AZ as instance. Cross AZ EFA comms is not supported and validating these will make sure client is only configured if needed, reducing any overhead. 2. Also updated the configure client function to check for EFA device and provider on the instance instead of hardcoding instance types for EFA.

allela-roy · 2025-09-30T09:52:44Z

@mayankgupta14 , LGTM.
However, can you make the cross-AZ check to happen first in verify_fsx_efa_compatibility() to avoid the timeouts?

mayankgupta14 · 2025-10-02T17:04:07Z

@mayankgupta14 , LGTM. However, can you make the cross-AZ check to happen first in verify_fsx_efa_compatibility() to avoid the timeouts?

ack, sure.

…avoid timeouts.

bluecrayon52 · 2025-10-14T19:51:41Z

@mayankgupta14 is this good to test or still being worked?

mayankgupta14 · 2025-10-15T01:29:19Z

@mayankgupta14 is this good to test or still being worked?

It's good to be tested and merged.

paragao

please review the EFA checks as using this LCS is blocking the deployment of compute nodes.

paragao · 2025-10-16T11:36:49Z

1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/mount_fsx.sh

+
+    # Check if instance has EFA drivers installed and configured
+    local efa_output
+    efa_output=$(fi_info -p efa 2>/dev/null)


this check is failing. I suggest you either use the full path /opt/amazon/efa/bin/fi_info or try to check for EFA using lsmod |grep efa (driver loaded) or lspci |grep EFA (PCI device exists).

mayankgupta14 · 2025-10-16T13:53:22Z

please review the EFA checks as using this LCS is blocking the deployment of compute nodes.

As discussed, the latest commit should address this.

paragao

I've ran the example on a working SLURM cluster with an EFA-enabled FSx Lustre file system. I confirm it works.

updated mount_fsx.sh to configure FSxL EFS client

8ee3915

KeitaW reviewed Sep 11, 2025

View reviewed changes

nghtm reviewed Sep 11, 2025

View reviewed changes

nghtm requested a review from paragao September 11, 2025 01:10

elgalu reviewed Sep 12, 2025

View reviewed changes

Addressing comment from Roy, moved cross-AZ check to happen first to …

7cea5ec

…avoid timeouts.

paragao changed the title ~~updated mount_fsx.sh to configure FSxL EFS client~~ updated mount_fsx.sh to configure FSxL EFA client Oct 3, 2025

paragao requested changes Oct 16, 2025

View reviewed changes

updated fi_info with absolute path to address command not found issue.

8d30cf9

paragao approved these changes Oct 16, 2025

View reviewed changes

paragao merged commit 59a0c1b into main Oct 16, 2025
4 checks passed

paragao deleted the fsx_efa_mayankpg branch October 16, 2025 15:43

		case "$instance_type" in
		p6-b200.\|p6e-gb200.\|p4d.\|p5.\|p5e.\|p5en.\|trn1.\|trn1n.\|c5n.\|c6gn.\|c6in.\|c7gn.\|g6.\|g6e.\|m5dn.\|m5n.\|m5zn.\|m6a.\|m6i.\|m6in.\|r5dn.\|r5n.\|r6in.\|x2gd.)

updated mount_fsx.sh to configure FSxL EFA client #849

updated mount_fsx.sh to configure FSxL EFA client #849

Uh oh!

Conversation

mayankgupta14 commented Sep 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mayankgupta14 Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KeitaW commented Sep 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elgalu commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elgalu commented Sep 12, 2025

Uh oh!

elgalu commented Sep 16, 2025

Uh oh!

allela-roy commented Sep 30, 2025

Uh oh!

mayankgupta14 commented Oct 2, 2025

Uh oh!

bluecrayon52 commented Oct 14, 2025

Uh oh!

mayankgupta14 commented Oct 15, 2025

Uh oh!

paragao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mayankgupta14 commented Oct 16, 2025

Uh oh!

paragao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

mayankgupta14 Sep 12, 2025 •

edited

Loading

elgalu commented Sep 12, 2025 •

edited

Loading