end-to-end gpu driver testing enhancement

[shivakunv]

TODO--

*1. *

• matrix:

• driver:
• • 535.183.06
• • 550.90.07
    An idea for a potential follow-up – instead of defining a matrix and spinning up one AWS instance per driver version, can we instead pass all driver versions as input to the test script and test all of them in sequence? For example, first install 535.183.06, and then upgrade to 550.90.07? This would also allow us to source the driver versions from the versions.mk file, instead of having to redefine and maintain that list here.

2
+name: CI
Let's rename this to End-to-end tests

3
+# Install the operator with usePrecompiled mode set to true
Remove this comment as it is not accurate.

4

• AWS_SESSION_TOKEN: ${{ secrets.AWS_SESSION_TOKEN }}
  I don't believe AWS_SESSION_TOKEN is needed with current holodeck implementation. Let's remove.

5
We can simply wait for the nvidia-driver pod to be ready
kubectl wait -n ${TEST_NAMESPACE} --for=condition=Ready pod -l app=nvidia-driver-daemonset --timeout 10m
If successful, then wait for the validator pod to be ready (this means that the rest of the pods are healthy):
kubectl wait -n ${TEST_NAMESPACE} --for=condition=Ready pod -l app=nvidia-operator-validator --timeout 2m
If either of these commands fails, capture the state of all pods in the operator namespace, by running kubectl get pods -n ${TEST_NAMESPACE} , and also capture some logs so we can debug.
This will reduce the amount of logs emitted during the test. Right now, we print out all pods every 5 seconds so it is very unreadable

[shivakunv]

@cdesiniotis PTAL