[datadog-agent] CiliumNetworkPolicy issues due to upstream Cilium issue affecting tunnelled Cilium deploys

[wparr-circle]

Recently when attempting to move from a self-managed CiliumNetworkPolicy to the CiliumNetworkPolicy managed by the helm chart in this repository - we discovered that it had started causing drops for ports 8125 UDP and 8126 TCP.

This was odd, given the presence of the following entries in the CiliumNetworkPolicy

- description: Ingress for dogstatsd
  endpointSelector:
    matchLabels:
      app: datadog
  ingress:
  - fromEndpoints:
    - {}
    toPorts:
    - ports:
      - port: "8125"
        protocol: UDP
- description: Ingress for APM trace
  endpointSelector:
    matchLabels:
      app: datadog
  ingress:
  - fromEndpoints:
    - {}
    toPorts:
    - ports:
      - port: "8126"
        protocol: TCP

There seems to be a similar issue previously reported #428

We managed to track this down to an upstream cilium issue which was summarised to

using overlay routing, then a client accesses a remote pod via hostPort with remote-node source identity (due to Masquerading when the client traffic leaves its own node). And the accessed pod's ingress policy needs to allow for this identity.