mergegateway broken in 1.0.2

[Uburro]

Description:
when i have upgrade envoy gateway from 1.0.1 to 1.0.2 i saw that envoy gateway operator does not merge gateway anymore. And see a lot of deployment when i created kind: Gateway. Rolled back deployment fixed this problem.

Environment:
kubernetes: v1.27.3
Envoy Gateway 1.0.2

[arkodg]

hey @Uburro can you share the errors from envoy gateway logs as well as the config ?

[Uburro]

hey @Uburro can you share the errors from envoy gateway logs as well as the config ?

Hi
cm

data:
  envoy-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    extensionApis:
      enableEnvoyPatchPolicy: true
    gateway:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    logging:
      level:
        default: info
    provider:
      type: Kubernetes
kind: ConfigMap

Enoyproxy

  logging:
    level:
      default: warn
  mergeGateways: true
  provider:
    kubernetes:
      envoyDeployment:
        container:
          image: company.cloud/mirror/io/docker/envoyproxy/envoy:distroless-v1.29.2
          resources:
            limits:
              memory: 1Gi
            requests:
              cpu: 350m
              memory: 1Gi
        patch:
          value:
            spec:
              template:
                spec:
                  containers:
                  - name: envoy
                    ports:
                    - containerPort: 10443
                      name: https
                      protocol: TCP
                    - containerPort: 10080
                      name: http
                      protocol: TCP
        pod:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: node-role.company.com/msp-infra
                    operator: DoesNotExist
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                  - key: envoy-gateway
                    operator: In
                    values:
                    - private
                topologyKey: kubernetes.io/hostname
          labels:
            envoy-gateway: private
        replicas: 2
      envoyService:
        externalTrafficPolicy: Local
        patch:
          value:
            spec:
              ipFamilies:
              - IPv4
              ports:
              - name: https
                port: 443
                protocol: TCP
                targetPort: 10443
              - name: http
                port: 80
                protocol: TCP
                targetPort: 10080
        type: LoadBalancer
    type: Kubernetes
  telemetry:
    accessLog:
      settings:
      - format:
          json:
            authority: '%REQ(:AUTHORITY)%'
            bytes_received: '%BYTES_RECEIVED%'
            bytes_sent: '%BYTES_SENT%'
            downstream_local_address: '%DOWNSTREAM_LOCAL_ADDRESS%'
            downstream_remote_address: '%DOWNSTREAM_REMOTE_ADDRESS%'
            duration: '%DURATION%'
            host: '%UPSTREAM_HOST%'
            message: '%LOCAL_REPLY_BODY%'
            method: '%REQ(:METHOD)%'
            path: '%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%'
            protocol: '%PROTOCOL%'
            request_id: '%REQ(X-REQUEST-ID)%'
            requested_server_name: '%REQUESTED_SERVER_NAME%'
            response_code: '%RESPONSE_CODE%'
            response_flags: '%RESPONSE_FLAGS%'
            upstream_cluster: '%UPSTREAM_CLUSTER%'
            upstream_host: '%UPSTREAM_HOST%'
            upstream_local_address: '%UPSTREAM_LOCAL_ADDRESS%'
            user_agent: '%REQ(USER-AGENT)%'
            x_forwarded_for: '%REQ(X-FORWARDED-FOR)%'
          type: JSON
        sinks:
        - file:
            path: /dev/stdout
          type: File

Also i see when merge gateway works correct label gateway.envoyproxy.io/owning-gatewayclass in deployment
When merge gateway does not work just

        gateway.envoyproxy.io/owning-gateway-name: spark-gateway-sparkcluster-e0tfrazvr7sang80t3
        gateway.envoyproxy.io/owning-gateway-namespace: sparkcluster-e0tfrazvr7sang80t3

[arkodg]

it looks like merge gateway didnt kick in, any negative status in GatewayClass or Gateway ?

[Uburro]

just No addresses have been assigned to the Gateway

GatewayClass

  conditions:
  - lastTransitionTime: "2024-07-01T12:34:40Z"
    message: Valid GatewayClass
    observedGeneration: 2
    reason: Accepted
    status: "True"
    type: Accepted

Gateway

status:
  conditions:
  - lastTransitionTime: "2024-07-01T12:38:04Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2024-07-01T12:38:04Z"
    message: No addresses have been assigned to the Gateway
    observedGeneration: 1
    reason: AddressNotAssigned
    status: "False"
    type: Programmed
  listeners:
  - attachedRoutes: 2
    conditions:
    - lastTransitionTime: "2024-07-01T12:38:04Z"
      message: Sending translated listener configuration to the data plane
      observedGeneration: 1
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2024-07-01T12:38:04Z"
      message: Listener has been successfully translated
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-07-01T12:38:04Z"
      message: Listener references have been resolved
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: http
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
    - group: gateway.networking.k8s.io
      kind: GRPCRoute
  - attachedRoutes: 0
    conditions:
    - lastTransitionTime: "2024-07-01T12:38:04Z"
      message: Sending translated listener configuration to the data plane
      observedGeneration: 1
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2024-07-01T12:38:04Z"
      message: Listener has been successfully translated
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-07-01T12:38:04Z"
      message: Listener references have been resolved
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: https
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: TLSRoute

[Uburro]

but EXTERNAL-IP assigned to kind: service

[Uburro]

intersting, when i have deleted all my gateway and deploy it again (in version 1.0.2) - all of them work ok. Where envoy gateway store metadata about do i need merged gateway or not? i see this one but it for pathing labels of deployemtn and services

[shawnh2]

Hi @Uburro, curious about your upgrade steps ?

Did you first remove existing EnvoyGateway ? and then reinstall it ? If is, then is related to #2637. The lots of deployment you saw, are probably got created right after the uninstall steps.

[Uburro]

Hi @shawnh2. I deployed through fluxcd in helm release controller. It similar helm upgrade -i ....
Do i need delete envoy gateway after that install int when i want to deploy new version of operator?