[Epic] CA lifecycle management

[anmazzotti]

This epic is the result of #1604 investigation.

Solves the following problems:

• There is no cacerts lifecycle management in Elemental. elemental-system-agent, rancher-system-agent, will stop working on all installed machines once the cacerts is renewed. This also affects the elemental-register in a very similar way.

Oct 03 09:55:54 test-e5331e3b-1e1b-4ce7-b080-235ed9a6d07c rancher-system-agent[82421]: time="2024-10-03T09:55:54Z" level=info msg="Starting remote watch of plans"
Oct 03 09:55:54 test-e5331e3b-1e1b-4ce7-b080-235ed9a6d07c rancher-system-agent[82421]: time="2024-10-03T09:55:54Z" level=info msg="Initial connection to Kubernetes cluster failed with error Get \"https://172.18.0.2.sslip.io/version\": x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"elemental-selfsigned-ca\"), removing CA data and trying again"
Oct 03 09:55:54 test-e5331e3b-1e1b-4ce7-b080-235ed9a6d07c rancher-system-agent[82421]: time="2024-10-03T09:55:54Z" level=fatal msg="error while connecting to Kubernetes cluster with nullified CA data: Get \"https://172.18.0.2.sslip.io/version\": x509: certificate signed by unknown authority"

• There is no easy way for users to use a public CA for Rancher, in combination with Elemental. This is a scenario where the cacerts will be empty, and the Elemental agents will have nothing to strictly validate. The workaround is to manually populate the Rancher's cacerts setting with the public CA certificate.

• Since the elemental-operator reads the cacerts and server-url Rancher settings to populate the MachineRegistration's caCert and url values, it is not possible for Elemental users to use different ingresses (that Rancher is not aware of). This is however needed if for example users would like to route different machine pools to different registration load balancers.

Issues (in order of priority):

1. [CA lifecycle] Allow agent-tls-mode setting on MachineRegistration elemental-operator#858
2. [CA lifecycle] Allow caCert override on MachineRegistration elemental-operator#859
3. [CA lifecycle] Allow server-url override on MachineRegistration elemental-operator#860
4. [CA lifecycle] OEM partition snapshotting elemental-toolkit#2203
5. [CA lifecycle] Re-apply MachineRegistration on updates elemental-operator#861

[anmazzotti]

Since we have no way to effectively influence the rancher-system-agent, the epic makes way less sense, as any registration-overridden setting will only apply to the elemental-system-agent. This needs to be solved first.
See: rancher/rancher#47386