New ha-policy feature proposal

[wangshu3000]

Hi

I have a feature proposal that i would like to get some feedback.

Here is our case: We are trying to deploy the artemis cluster on kuberentes, we need to deploy the shared-store ha-policy to meet our HA requirement. We need to make sure all the data are still accessible under an az outage.
We need to deploy at least 6 brokers in below pattern.
Group0: broker0 (az1) live -- broker1 (az2) backup share same volume
Group1: broker2 (az3) live -- broker3 (az1) backup share same volume
Group2: broker4 (az2) live -- broker5 (az3) backup share same volume
All the 6 brokers are in one cluster, work as a mesh (network of brokers), so messages produce to any broker, can be consumer from any broker as well.

The shared store ha-policy is a native feature of artemis mq, but not supported by the operator. So I would like to propose this new feature to support the ha-policy deployment by operator.

Below is the proposed change details.

1. Update CRD to add haPolicy section under deployPlan
2. Change the stateful set reconcile logic, to deploy multiple statefulsets for the CR. One pair of brokers will be a statefulset, so 6 brokers, we will create 3 statefulsets.
3. Create pvc for each sts rather than for each pod.
4. Configure the sts to use the pvc to share the same volume. And not generate the pvctemplate section.
5. Generate svc for each acceptor/console once, name will be the same as normal deployment, only change the selector part to select the correct pod
6. Generate ing for each acceptor/console once, name will be the same as normal deployment
7. Generate global level resources once, including hdls/ping svc, ing, secret etc.
8. Update init container logic to add the ha-policy configurations to the broker.xml file
9. Make sure data can be migrated when scale down or change policy
10. Update the status conditions of the CR accordingly

I have finished an initial version to deploy shared store ha-policy cluster. And tested in my own kubernetes.
Not sure if these changes make sense. If it's reasonable, i would like to contribute these code back to this repo.

Thanks.

[brusdev]

@wangshu3000 thanks for sharing your proposal. It is interesting, could you share further details about your use case? How the availability zones are connected? What storage are you using to share the same PV across availability zones? What settings are you using to ensure that a broker pod is deployed in the desired availability zone?

[wangshu3000]

@brusdev Thanks for your reply, i think we can use node affinity, node anti-affinity and pod TopologySpreadConstraint to control the placement of the pod.
We deploy the cluster on EKS, the network is controlled by AWS, and we are using the AWS EFS as the storage. and share it cross 2 brokers from 2 AZs (1 live and 1 backup). When the live broker is down, the backup can take over very fast. Even when the kubernetes can't start new broker, it'll still make sure the cluster has enough broker to take traffic.
Not sure if this deployment make sense.

The deployment can either be:
Group0: broker0 (az1) live -- broker1 (az2) backup share same volume
Group1: broker2 (az3) live -- broker3 (az1) backup share same volume
Group2: broker4 (az2) live -- broker5 (az3) backup share same volume
Or
Group0: broker0 (az1) live -- broker1 (az2) backup share same volume
Group1: broker2 (az2) live -- broker3 (az3) backup share same volume
Group2: broker4 (az3) live -- broker5 (az1) backup share same volume
Thanks.