[WIP] Add RAPIDS Nightly to GPU CI

[praateekmahajan]

Description

Usage

# Add snippet demonstrating usage

Checklist

• I am familiar with the Contributing Guide.
• New or Existing tests cover these changes.
• The documentation is up to date with these changes.

[sarahyurick]

We might need to check if the nightly can use the same caching logic as stable, or if nightly should just always build a fresh non-cached container.

[praateekmahajan]

+1, wanted to flag this explicitly as we do some curator-update optimization above, and I want to ensure if branching doesn't mess with it.
cc @ko3n1g

[ko3n1g]

Do these optimizations come from the base-image?

[ko3n1g]

I mean, if they do even if the base-image tag is unchanged b/c they overwrite it we should extend this build-workflow so that we can force pull the base image. In all other cases we don't need to be worried about caching i believe

[ko3n1g]

Ok so here my suggestion:

• We run the gpuci workflow on nightly schedule (via github workflows cron trigger)
• We use use-cache: false to make sure we build from scratch with latest dependencies (merge ci: Bump _build_container workflow #448 before)
• It doesn't matter whether we build two containers or one with two venvs. Initially I thought it would be easier for debugging we if we had a single container, but two containers (like you already prepared) have the benefit that size / container is smaller which come in handy next year
• Tip: if: ${{ github.event.label.name == 'gpuci' || github.ref == 'refs/heads/main' }} needs to be changed to if: ${{ github.event.label.name == 'gpuci' || github.ref == 'refs/heads/main' || github. event_name == 'schedule' }}