E2E Test Foundations

[gflarity]

What type of PR is this?

/kind test

What this PR does / why we need it:

This PR introduces E2E tests for grove. It uses k3d which in turn leverages k3s, I lightweight yet compliant Kubernetes distribution.These were selected for their performance and ability to limit the amount of memory per worker/agent node.

The goal was use pure Go libraries as much as possible E2E.

Special notes for your reviewer:

• I was able to get all Gang-Scheduling tests from Grove Multi-Node Test Plan working, but I removed most of them to keep this PR managable.
• E2E tests are currently disabled by default. I'd like to discuss when we want these to run before I enable them as the full tests take about 15m.
• Currently all E2E tests run KAI and assume it's running, the future when there are more scheduling options we can change this.

Does this PR introduce a API change?

NONE

Additional documentation e.g., enhancement proposals, usage docs, etc.:

You'll need to run prepare charts at least once:
`./operator/hack/prepare-chart.sh`

For best results, run go test with -v and -p 1: 
`go test -v -tags=e2e -p 1 ./e2e_testing/...`

Here's sample of test results:

[gflarity]

Thanks @shayasoolin, I've integrated your feedback. Please take another look.

[unmarshall]

@gflarity can you please rebase your PR so that i can look at the code from within the IDE, without a lot of errors?
Rebasing will resolve all the import package errors that are there in your PR (without rebase) due to migration to ai-dynamo

I have only partially looked at the code and i will anyways push the initial set of comments. There is a lot of code to still look at which i will do once you have rebased the PR.

[gflarity]

@gflarity can you please rebase your PR so that i can look at the code from within the IDE, without a lot of errors? Rebasing will resolve all the import package errors that are there in your PR (without rebase) due to migration to ai-dynamo

I have only partially looked at the code and i will anyways push the initial set of comments. There is a lot of code to still look at which i will do once you have rebased the PR.

np, done

[renormalize]

Thanks for the PR @gflarity! The framework looks great -
I've added minor nits.

[gflarity]

Thanks for the PR @gflarity! The framework looks great - I've added minor nits.

Thanks for the feedback @renormalize. I've implemented all the changes. I'll chat with @sanjaychatterjee and get this merged.

As discussed in the meeting, we can just merge this can address any outstanding feedback in follow up PRs since this doesn't affect the operator itself at all.