Prevent KafkaCluster from being stuck at ClusterReconciling on users' misconfigurations

[panyuenlau]

Problem Statement

When users configure listenersConfig in a way that it requires the server secret that contains the SSL certificates, but they neither provide the necessary secret nor tell the Koperator to create the secret, the KafkaCluster would be stuck at ClusterReconciling state and the users would be confused.

Relevant KafkaCluster configurations that would cause this behavior:

listenersConfig:
    externalListeners:
      - type: "ssl"
        name: "external"
        externalStartingPort: 19090
        containerPort: 9094
    internalListeners:
      - type: "plaintext"
        name: "plaintext"
        containerPort: 29092
        usedForInnerBrokerCommunication: true
      - type: "plaintext"
        name: "controller"
        containerPort: 29093
        usedForInnerBrokerCommunication: false
        usedForControllerCommunication: true

The above configurations would cause the Koperator to be stuck at this line:

https://github.com/banzaicloud/koperator/blob/master/pkg/resources/kafka/kafka.go#L610

Example logs:

{"level":"info","ts":"2023-05-11T15:08:16.700Z","msg":"A new resource was not found or may not be ready","controller":"KafkaCluster","controllerGroup":"kafka.banzaicloud.io","controllerKind":"KafkaCluster","KafkaCluster":{"name":"kafka-pl
aintext","namespace":"kafka"},"namespace":"kafka","name":"kafka-plaintext","reconcileID":"3b163d8b-502a-443b-a3f4-d4a49c91e59a","error":"server secret not ready: Secret \"kafka-plaintext-server-certificate\" not found"}

Proposed Solution

The KafkaCluster validation webhook should reject such KafkaCluster creation and/or update requests.

Alternatives Considered

N/A

Additional Context

N/A