Connection Timeout 504 Issue with Kuma, KIC, and Kong Gateway

[Frickson]

I am using Kuma, KIC, and Kong gateway for my application. Despite my current configurations, I still encounter a connection timeout 504 error when my application processes a task that takes more than 60 seconds. Here are the details:

Mesh Configuration

mesh-name: msbu

MeshTimeout Configuration:

apiVersion: kuma.io/v1alpha1
kind: MeshTimeout
metadata:
  name: timeout
  namespace: kuma-system
  labels:
    kuma.io/mesh: msbu
spec:
  targetRef:
    kind: Mesh
  from:
  - targetRef:
      kind: Mesh
    default:
      idleTimeout: 300s
      connectionTimeout: 300s
      http:
        requestTimeout: 300s
        streamIdleTimeout: 1h
        maxStreamDuration: 30m
        maxConnectionDuration: 30m
  to:
  - targetRef:
      kind: Mesh
    default:
      idleTimeout: 300s
      connectionTimeout: 300s
      http:
        requestTimeout: 300s
        streamIdleTimeout: 1h
        maxStreamDuration: 30m
        maxConnectionDuration: 30m

Kong Proxy Service Configuration:

apiVersion: v1
kind: Service
metadata:
  annotations:
    konghq.com/connect-timeout: "60000000"
    konghq.com/read-timeout: "60000000"
    konghq.com/retries: "3"
    konghq.com/write-timeout: "60000000"
    meta.helm.sh/release-name: kong
    meta.helm.sh/release-namespace: kong
  labels:
    app.kubernetes.io/instance: kong
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kong
    app.kubernetes.io/version: "3.6"
    enable-metrics: "true"
    helm.sh/chart: kong-2.38.0
  name: kong-proxy
  namespace: kong
spec:
  type: NodePort
  # More configuration below...

Kong Proxy Pod Environment Variable:

ENV KONG_UPSTREAM_KEEPALIVE_IDLE_TIMEOUT='300'

Issue:

After setting up all the above configurations, I still get a connection timeout 504 when my application processes a task that takes more than 60 seconds. I would appreciate any help in resolving this issue.

Thank you!

[StarlightIbuki]

@randmonkey Could you take a look?