[BUG] KSM: HPA metrics don't get standard labels applied

[shivshav]

Agent Environment

Deployed in Kubernetes using the official Datadog Helm chart.

Versions:

• Helm chart: 3.67.5
• Agent: 7.54.0
• Cluster Agent: 7.54.0

Our whole kubeStateMetricsCore configuration from our usage of the Datadog Helm chart

...
    kubeStateMetricsCore:
      enabled: true
      # Don't run on main agent daemonset pods to keep resource usage stable across pods
      useClusterCheckRunners: true
      collectVpaMetrics: true
      # Common labels found on most K8S resources that we'd like as tags in DD
      # See https://docs.datadoghq.com/integrations/kubernetes_state_core/?tab=helm#default-labels-as-tags for labels
      # already collected
      ksm_common_label_map: &ksm_common_label_map
        # Our custom labels
        app: leafly_app
        notifications_leafly_io_slack: notifications_route_slack
        notifications_leafly_io_opsgenie: notifications_route_opsgenie
        process: leafly_process
        release: leafly_release
        # External labels we wanna collect
        k8s_app: kube_system_app  # usually provided by kOps on manifests it manages
      labelsAsTags:
        container: *ksm_common_label_map
        cronjob: *ksm_common_label_map
        daemonset: *ksm_common_label_map
        deployment: *ksm_common_label_map
        endpoint: *ksm_common_label_map
        horizontalpodautoscaler: *ksm_common_label_map
        ingress: *ksm_common_label_map
        job: *ksm_common_label_map
        node:
          kops_k8s_io_instancegroup: kops_instancegroup
          leafly_node_util: util_node
        pdb: *ksm_common_label_map
        persistentvolumeclaim: *ksm_common_label_map
        pod: *ksm_common_label_map
        replicaset: *ksm_common_label_map
        service: *ksm_common_label_map
        statefulset: *ksm_common_label_map
...

Describe what happened:
Our KSM-based HPA metrics do not have the default labels attached to them unlike other resources (see screenshots below).

What our tags look like for a kubernetes_state.hpa. metric (kubernetes_state.hpa.max_replicas in this case)

What our tags look like for another kubernetes_state. metric that is working as expected (kubernetes_state.deployment.replicas_ready in this case)

Describe what you expected:
I expect most of the kubernetes_state.hpa.* metrics to have the following tags attached to them with no configuration on our end based on the documentation here (I can confirm the labels exist on our resources)

• kube_app_name
• kube_app_instance
• kube_app_version
• kube_app_component
• kube_app_part_of
• kube_app_managed_by
• helm_chart
• env
• service
• version

Steps to reproduce the issue:

1. Deploy Datadog helm chart to Kubernetes cluster with kubeStateMetricsCore values config noted above
2. Deploy HPA resource that utilizes standard K8S labels and/or Datadog Unified Service Tagging labels
3. Observe metrics emitted by Datadog do not have the aforementioned tags attached

Additional environment details (Operating System, Cloud provider, etc):

Potential Fix?

I noticed the defaultLabelJoins function here does not include kube_horizontalpodautoscaler_labels in the returned map, which is what KSM calls the label metric it emits. So maybe that's all that needs to happen for this issue to be resolved? But not certain.