Replace TAS plugin with Kueue #3534
Conversation
annuay-google
added
the
release-improvements
annuay-google
added
release-key-new-features
| - id: workload-manager-install | ||
| source: modules/management/kubectl-apply | ||
| use: [a3-ultragpu-cluster] | ||
| settings: | ||
| kueue: | ||
| install: true | ||
| version: v0.10.0 | ||
| config_path: $(vars.kueue_configuration_path) | ||
| config_template_vars: | ||
| num_gpus: $(a3-ultragpu-pool.total_gpu_count) |
There was a problem hiding this comment.
This is super, so just to confirm if the nodepool size is changed via the toolkit, this config will be updated ?
There was a problem hiding this comment.
Yes- this number is fetched as an output of the node pool module. Calculated as allocatable_gpu_per_node * static_node_count. This is done so that user doesn't have to pass a redundant input (num_gpus)
Deleting this as well, since we have an n-node test now |
Deleted |
| @@ -50,6 +50,11 @@ output "allocatable_gpu_per_node" { | |||
| value = local.allocatable_gpu_per_node | |||
| } | |||
|
|
|||
| output "total_gpu_count" { | |||
There was a problem hiding this comment.
Wondering if this should be static_gpu_count. Might have implications down the road for configuring Kueue with DWS Flex, since DWS Flex + Kueue could be configured with the total maximum scale.
What?
Why?
Kueue is the officially supported job scheduling utility for A3 Ultra
Testing
Ran Jobset based NCCL test, verified bandwidth figures
Submission Checklist
NOTE: Community submissions can take up to 2 weeks to be reviewed.
Please take the following actions before submitting this pull request.