Multiple installations of cloud-api-adaptor on the same cluster

[bpradipt]

In some scenarios there is a need to support multi-cloud pod VM creation from the same K8s cluster. For example, the K8s cluster could be on-premise, but it's creating peer-pod on AWS, Azure and IBM cloud.

Creating this tracker issue to discuss on how such a deployment model be supported?

cc @stevenhorsman @mkulke

[mkulke]

It probably makes sense to think first about what a good ux would be and make also reference this with the "profile" idea.

I imagine the best UX would be to offer multiple runtime classes, e.g. kata-remote-azure or kata-remote-gcp-large kata-remote-aws-gpumedium and a single CAA installation is owning those RCs and dispatching it to a provisioner and profile

[bpradipt]

Yeah different cloud-providers can be supported via different runtimeClasses

At a high level it may look like this

Kata configs

configuration-remote-azure.toml
...
[hypervisor.remote]
remote_hypervisor_socket = "/run/peerpod/hypervisor-azure.sock"
...

configuration-remote-aws.toml
...
[hypervisor.remote]
remote_hypervisor_socket = "/run/peerpod/hypervisor-aws.sock"
...

configuration-remote-ibmcloud.toml
...
[hypervisor.remote]
remote_hypervisor_socket = "/run/peerpod/hypervisor-ibmcloud.sock"
...

Then comes the CAA daemonset and it's associated configs (peer-pods-secret, peer-pods-cm and in future profiles).

May be different namespaces for each runtimeClass ?

[snir911]

I agree runtimeclasses would be the best UX, however, there's the use case of same cloud-provider but different accounts, IDK how commonly it's needed but i heard some interest in that.

Would it make sense to use current workflow and on top of it allow some kind of profiles? by default caa will use the global configuration unless profile is explicitly set to a pod (or something like that, maybe it's too flexible approach )

[mkulke]

hmm, should there be different socket for different cloud providers?

theoretically all the -remote runtimeclasses could carry only informational value. different kata-remote-* RC's would being handled by the same CAA instance, which uses it to dispatch to different profiles. In this sense a cloud-provider could be a field in a CAA profile: kata-remote-aws => { cloud-provider: aws, provider-config: { region: east-us, vpc: abc }}

[mkulke]

Would it make sense to use current workflow and on top of it allow some kind of profiles? by default caa will use the global configuration unless profile is explicitly set to a pod (or something like that, maybe it's too flexible approach )

this is being explore here: #2218

I think it would be good to make such a profile a first-class object, similar to e.g. how kubeconfig handles contexts, there can be a default context, but the context is a list of configurations

[bpradipt]

hmm, should there be different socket for different cloud providers?

I think each runtimeClass will need a specific CAA instance and hence different socket. No?

[mkulke]

hmm, should there be different socket for different cloud providers?

I think each runtimeClass will need a specific CAA instance and hence different socket. No?

not sure, is there a technical reason for that? can a single CAA instance handle multiple runtime classes?

[bpradipt]

hmm, should there be different socket for different cloud providers?

I think each runtimeClass will need a specific CAA instance and hence different socket. No?

not sure, is there a technical reason for that? can a single CAA instance handle multiple runtime classes?

We run the CAA with cloud-provider as an option. So I think at a minimum separate CAA instance will be needed per cloud-provider. Not sure how this restriction can be relaxed..

[mkulke]

We run the CAA with cloud-provider as an option. So I think at a minimum separate CAA instance will be needed per cloud-provider. Not sure how this restriction can be relaxed..

ah, I was musing about supporting multiple cloud providers in a single CAA instance.

[bpradipt]

Looking at it from a deployment perspective (with minimal code changes), following are some approaches that comes to my mind:

1. Use provider specific configmaps and secrets and run multiple daemonsets (one for each provider). This approach will use separate CAA daemonsets (pods), one per runtimeclass.
2. Add multiple provider specific configuration in the same configmap and secret and have separate containers for each provider in the same daemonset. This approach will use single CAA daemonset (pod), but multiple containers - one per runtimeclass.