Ingress missing with multiple nodes?

[lordigon]

I'm trying to create a two node Artemis MQTT broker with your latest operator version 1.0.13

apiVersion: broker.amq.io/v1beta1
kind: ActiveMQArtemis
metadata:
  name: exoo
spec:
  deploymentPlan:
    size: 2
    image: placeholder
    persistenceEnabled: true
    messageMigration: true
  acceptors:
    - name: mqtt-acceptor
      protocols: MQTT
      port: 61616
      expose: false
      sslEnabled: false
    - name: mqtt-tls-acceptor
      protocols: MQTT
      port: 61617
      expose: true
      sslEnabled: true
      needClientAuth: true
      verifyHost: false

but I cannot see any ingress entry related to the exposed acceptor. Is it the right behaviour?

[gtully]

not if your are deploying to kube. If you are deploying on openshift then there will be a route where an ingress would be.

[lordigon]

@gtully What is the best practice to use in this case to expose that acceptor outside the cluster?

[brusdev]

@lordigon I recentily added a tutorial to connect to the broker from external mqtt clients: https://github.com/artemiscloud/activemq-artemis-operator/blob/main/docs/tutorials/external_mqtt_clients.md

[lordigon]

Thx @brusdev for this info.
I think could be useful also to add examples on how to manage connections outside the cluster when there are more than one replica of the Broker.
What happen to your example if you set size equal to 2 to your CRD (that is my initial question)?
Using Artemis on k8s is useful if we can leverage k8s native features like autoscaling but I don't know how an MQTT client outside the cluster, both producer and consumer side, should manage the connections in this use case.

[brusdev]

@lordigon the operator creates only independent Ingress and Service resources for every acceptor of every broker pod. You could create a LoadBalancer Service to spread connections across the broker pods but the optimal solution would be to have both producer and consumer of the same topic connected to the same broker pod. You could define a connection router with the broker properties to pratition the MQTT clients having both producer and consumer of the same topic connected to the same broker pod.

[lordigon]

@brusdev regarding this sentence "the operator creates only independent Ingress and Service resources for every acceptor of every broker pod" with the CRD I wrote in my first comment on minikube and AWS

I got these svc

NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE
exoo-hdls-svc                  ClusterIP   None             <none>        8161/TCP,61616/TCP   89s
exoo-mqtt-acceptor-0-svc       ClusterIP   10.96.79.115     <none>        61616/TCP            89s
exoo-mqtt-acceptor-1-svc       ClusterIP   10.103.16.247    <none>        61616/TCP            89s
exoo-mqtt-tls-acceptor-0-svc   ClusterIP   10.107.165.188   <none>        61617/TCP            89s
exoo-mqtt-tls-acceptor-1-svc   ClusterIP   10.98.238.205    <none>        61617/TCP            89s
exoo-ping-svc                  ClusterIP   None             <none>        8888/TCP             89s

and no ingress.
So it is a bug or I misunderstood what you said?
Regarding instead "to have both producer and consumer on the same broker pod": I though that a client shouldn't know how many nodes an Artemis cluster is made of pointing to a single hostname so that a scale in or scale out doesn't affect estabilished client connections.

[gtully]

those services point to a single instance of the stateful set. and that is intentional.

I think your use case matches something I have been been working on to operationalise. There is a proof of concept in this test:
https://github.com/artemiscloud/activemq-artemis-operator/blob/main/controllers/activemqartemis_pub_sub_scale_test.go#L51

There is no route, b/c all the clients are in cluster, but the concepts are the same once the load balancing service in in play.
https://github.com/artemiscloud/activemq-artemis-operator/blob/main/controllers/activemqartemis_pub_sub_scale_test.go#L224

The idea is to allow publishers to connect to any broker, such that a publisher has the availability of the cluster. However consumers are partitioned such that we can scale and have the the availability of their partition. The partitioning uses roles matched to broker ordinals.
I appreciate that this is currently quite complex with all of the relevant config in brokerProperties and I would like to hide all this behind a broker-provisioning-cr. then have an app-onboarding-cr to capture the addresses/producers/consumers.

There is lots in this, but I would love to have your input to the process if you think your use case is a good match.
I am currently working on: https://github.com/artemiscloud/activemq-artemis-operator/blob/main/controllers/activemqartemis_pub_sub_scale_test.go#L370

[gtully]

that is a concrete example of what @brusdev suggested above.

[lordigon]

@brusdev do you have an example on how a connection router can be configured with a cluster with 2 replicas?