Add new project proposal to describe nvlink + topology aware scheduling

[ecolternv]

Description

Add project design document for nvlink + topology aware scheduling support

Issue #206

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[RyaliNvidia]

This is a more general question regarding the feature, but once the feature is there, how will you verify that NVLink is performing as expected? What kind of tests can you do to verify that NVLink is actually being used and whether it is performing as well as expected? It could be that there isn't an issue with the implementation and that NVLink is not performing well, but it would be nice to know that.

[RyaliNvidia]

Is NVLink a feature we want users to opt into so that even if the feature is there, they can specify not to use it? This is because they might want to build confidence that NVLink is improving their performance so they would want to run to side by side tests with NVLink there and not.

This way, we could get a KPI on how many people are switching to using NVLink and the usefulness of this feature once deployed

[RyaliNvidia]

Another thing to consider is if we can change the resources list to show the zones/racks/whatnot visually so users know why their workflow isn't scheduling even though there are resources available.

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA.github.io/OSMO/pr-preview/pr-211/
Built to branch gh-pages/documentation at 2026-01-26 21:48 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[ecolternv]

This is a more general question regarding the feature, but once the feature is there, how will you verify that NVLink is performing as expected? What kind of tests can you do to verify that NVLink is actually being used and whether it is performing as well as expected? It could be that there isn't an issue with the implementation and that NVLink is not performing well, but it would be nice to know that.

Yeah, we need a more comprehensive test workflow that is sensitive to network speed. Perhaps we can use https://github.com/NVIDIA/nccl-tests instead of doing model inference/training.

I added an open item for this

[ecolternv]

Another thing to consider is if we can change the resources list to show the zones/racks/whatnot visually so users know why their workflow isn't scheduling even though there are resources available.

Good idea, but it will take some design to do this right. I've added an open item for this in the topology-aware-scheduling doc

[ecolternv]

Is NVLink a feature we want users to opt into so that even if the feature is there, they can specify not to use it? This is because they might want to build confidence that NVLink is improving their performance so they would want to run to side by side tests with NVLink there and not.

This way, we could get a KPI on how many people are switching to using NVLink and the usefulness of this feature once deployed

You can opt not to use it by setting NCCL_MNNVL_ENABLE=0. In that case the compute domain just wont be used, which shouldn't cause any problems as far as I'm aware.