Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions charts/agentmesh/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,3 +168,9 @@ Mesh size and state come from the control plane
`agentmesh_control_plane_enrolled_nodes{role,state}`); the router adds its live view
(`agentmesh_router_authenticated_peers`, `agentmesh_router_auth_handshakes_total{result}`)
alongside libp2p's own relay and connection metrics.

## Several routers (`router.replicaCount` > 1)

With more than one replica the chart leaves out `--external-addr`, so each router announces the addresses it listens on: its own pod IP, for TCP and QUIC. A router is identified by its peer id, and the shared `-router` Service would send a dial to whichever pod it likes (the handshake then fails with "peer id mismatch"). The router puts its address in its lease, which it sends on its first sync with the control plane after it starts and renews from then on. With one replica, or with `router.externalAddrs` / `router.hostPort` set, nothing changes.

Change `replicaCount` through Helm (`helm upgrade --set router.replicaCount=N`). `kubectl scale` does not render the template again, so the routers keep the address the chart rendered before.
3 changes: 3 additions & 0 deletions charts/agentmesh/templates/router-statefulset.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,9 @@ spec:
{{- else if .Values.router.hostPort }}
- "--external-addr=/ip4/$(HOST_IP)/tcp/{{ .Values.router.hostPort }}"
- "--external-addr=/ip4/$(HOST_IP)/udp/{{ .Values.router.hostPort }}/quic-v1"
{{- else if gt (int .Values.router.replicaCount) 1 }}
# No --external-addr: each router announces the addresses it listens on, which are its own pod IP (TCP and QUIC).
# The shared Service name would send a dial to whichever pod it likes ("peer id mismatch").
{{- else }}
- "--external-addr=/dns4/{{ include "agentmesh.fullname" . }}-router/tcp/{{ .Values.router.service.port }}"
{{- end }}
Expand Down
26 changes: 26 additions & 0 deletions charts/agentmesh/tests/router-statefulset_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,32 @@ tests:
path: spec.template.spec.containers[0].args
content: --external-addr=/ip4/$(HOST_IP)/tcp/4501

- it: with several replicas announces no external address, so each router announces its own pod IP
set:
router.replicaCount: 2
documentSelector:
path: kind
value: StatefulSet
asserts:
- notContains:
path: spec.template.spec.containers[0].args
content: --external-addr=/dns4/agentmesh-router/tcp/4501
- isNull:
path: spec.template.spec.containers[0].env

- it: externalAddrs still wins with several replicas
set:
router.replicaCount: 2
router.externalAddrs:
- /dns4/x/tcp/1
documentSelector:
path: kind
value: StatefulSet
asserts:
- contains:
path: spec.template.spec.containers[0].args
content: --external-addr=/dns4/x/tcp/1

- it: keeps the key on a PVC, not emptyDir
documentSelector:
path: kind
Expand Down
1 change: 1 addition & 0 deletions site/content/docs/guides/kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,7 @@ which is meant for development clusters.

For the router, `router.hostPort: 4501` binds TCP and UDP port 4501 on the
Kubernetes node where the router pod runs, and announces that node's IP.
With more than one router replica, each router announces its own pod IP.
`router.externalAddrs` replaces the announced addresses, for example with a
fixed DNS name:

Expand Down
Loading