test: unit tests for idleTimeoutSeconds config per worker groups

[alimaazamat]

Why are these changes needed?

V2 Autoscaling allows for configuring idleTimeoutSeconds per worker group, which needs added unit + e2e testing
EDIT: e2e testing already done in #2725 issue should've been closed

Related issue number

#2561

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[marosset]

LGTM

[andrewsykim]

@ryanaoleary can you review since you added the feature?

[ryanaoleary]

LGTM, I tested some of these cases manually and it looks correct.

[andrewsykim]

@alimaazamat the e2e buildkite failure seems related? Can you take a look https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/11598#019a7efb-eeee-48fd-8614-6bd5d5311f4e

Triggered a retry incase it's a flake

[alimaazamat]

@alimaazamat the e2e buildkite failure seems related? Can you take a look https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/11598#019a7efb-eeee-48fd-8614-6bd5d5311f4e

@andrewsykim Yeah I reran it already previously and it did fail again. Because this are just unit tests I thought it wouldn't impact e2e?

[alimaazamat]

Actually I think I found the issue:
Seems like part2 autoscaler tests are using env var only and not the spec version.

[alimaazamat]

@andrewsykim
In buildkite the only errors I'm seeing are reconcillation errors where worker pod status is failed during teardown. https://github.com/alimaazamat/kuberay/blob/idleTimeoutSecondsTests/ray-operator/controllers/ray/raycluster_controller.go#L173 is the only time we call our ValidateRayClusterSpec changes and if it is invalid, the controller would return immediately and wouldn't even hit the errors? Let me know if my thinking is on the wrong track.