Security Agent pod in CrashLoopBackOff when disabled by values

[TheBlackMini]

Describe what happened:
Deployment using helm chart for 3.71.1 and the deployment fails due to the security-agent being stuck in CrashLoopBackOff

2024-09-16 07:17:54 UTC | SECURITY | INFO | (pkg/util/log/log.go:831 in func1) | Found 'HTTP_PROXY' env var, using it for the Agent proxy settings
2024-09-16 07:17:54 UTC | SECURITY | INFO | (pkg/util/log/log.go:831 in func1) | Found 'HTTPS_PROXY' env var, using it for the Agent proxy settings
2024-09-16 07:17:54 UTC | SECURITY | INFO | (pkg/util/log/log.go:831 in func1) | Found 'NO_PROXY' env var, using it for the Agent proxy settings
2024-09-16 07:17:54 UTC | SECURITY | WARN | (pkg/util/log/log.go:871 in func1) | Agent configuration relax permissions constraint on the secret backend cmd, Group can read and exec
2024-09-16 07:17:54 UTC | SECURITY | INFO | (pkg/util/log/log.go:831 in func1) | 3 Features detected from environment: kubernetes,cri,containerd
2024-09-16 07:17:54 UTC | SECURITY | WARN | (pkg/config/model/viper.go:192 in checkKnownKey) | config key fips.enabled is unknown
2024-09-16 07:17:54 UTC | SECURITY | INFO | (pkg/config/setup/config.go:1409 in func1) | Found 'HTTP_PROXY' env var, using it for the Agent proxy settings
2024-09-16 07:17:54 UTC | SECURITY | INFO | (pkg/config/setup/config.go:1409 in func1) | Found 'HTTPS_PROXY' env var, using it for the Agent proxy settings
2024-09-16 07:17:54 UTC | SECURITY | INFO | (pkg/config/setup/config.go:1409 in func1) | Found 'NO_PROXY' env var, using it for the Agent proxy settings
2024-09-16 07:17:54 UTC | SECURITY | WARN | (pkg/config/model/viper.go:192 in checkKnownKey) | config key use_proxy_for_cloud_metadata is unknown
2024-09-16 07:17:54 UTC | SECURITY | INFO | (pkg/security/utils/hostname.go:76 in GetHostnameWithContextAndFallback) | Hostname is:
2024-09-16 07:17:54 UTC | SECURITY | INFO | (subcommands/runtime/command.go:664 in StartRuntimeSecurity) | Datadog runtime security agent disabled by config
2024-09-16 07:17:54 UTC | SECURITY | INFO | (pkg/security/utils/hostname.go:76 in GetHostnameWithContextAndFallback) | Hostname is:
2024-09-16 07:17:54 UTC | SECURITY | INFO | (comp/core/workloadmeta/store.go:99 in start) | workloadmeta store initialized successfully
2024-09-16 07:17:54 UTC | SECURITY | INFO | (subcommands/start/command.go:267 in RunAgent) | All security-agent components are deactivated, exiting
2024-09-16 07:17:54 UTC | SECURITY | INFO | (pkg/util/containerd/containerd_util.go:175 in connect) | Connected to containerd - Version 1.6.31+bottlerocket/***********************************4a521
2024-09-16 07:17:55 UTC | SECURITY | INFO | (comp/core/workloadmeta/store.go:537 in startCandidates) | workloadmeta collector "containerd" started successfully
2024-09-16 07:17:55 UTC | SECURITY | INFO | (comp/core/workloadmeta/store.go:540 in startCandidates) | workloadmeta collector "docker" could not start. error: component workloadmeta-docker is disabled: Agent is not running on Docker
2024-09-16 07:17:55 UTC | SECURITY | INFO | (comp/core/workloadmeta/store.go:540 in startCandidates) | workloadmeta collector "cloudfoundry-container" could not start. error: component workloadmeta-cloudfoundry-container is disabled: Agent is not running on CloudFoundry
2024-09-16 07:17:55 UTC | SECURITY | INFO | (comp/core/workloadmeta/store.go:540 in startCandidates) | workloadmeta collector "podman" could not start. error: component workloadmeta-podman is disabled: Podman not detected
2024-09-16 07:17:55 UTC | SECURITY | INFO | (comp/core/workloadmeta/store.go:540 in startCandidates) | workloadmeta collector "ecs" could not start. error: component workloadmeta-ecs is disabled: Agent is not running on ECS EC2
2024-09-16 07:17:55 UTC | SECURITY | INFO | (pkg/util/kubernetes/kubelet/kubelet_client.go:289 in checkKubeletConnection) | Successful configuration found for Kubelet, using URL:
2024-09-16 07:17:55 UTC | SECURITY | INFO | (comp/core/workloadmeta/store.go:537 in startCandidates) | workloadmeta collector "kubelet" started successfully
2024-09-16 07:17:55 UTC | SECURITY | INFO | (comp/core/workloadmeta/store.go:540 in startCandidates) | workloadmeta collector "process-collector" could not start. error: collector process-collector is not enabled
2024-09-16 07:17:55 UTC | SECURITY | INFO | (comp/core/workloadmeta/store.go:540 in startCandidates) | workloadmeta collector "cloudfoundry-vm" could not start. error: component workloadmeta-cloudfoundry-vm is disabled: Agent is not running on CloudFoundry
2024-09-16 07:17:55 UTC | SECURITY | INFO | (pkg/api/security/security.go:194 in getClusterAgentAuthToken) | Using configured cluster_agent.auth_token
2024-09-16 07:17:55 UTC | SECURITY | INFO | (pkg/util/clusteragent/clusteragent.go:135 in init) | Successfully connected to the Datadog Cluster Agent 7.57.0+commit.e141ead
2024-09-16 07:17:55 UTC | SECURITY | INFO | (comp/core/workloadmeta/store.go:537 in startCandidates) | workloadmeta collector "kube_metadata" started successfully
2024-09-16 07:17:55 UTC | SECURITY | INFO | (comp/core/workloadmeta/store.go:537 in startCandidates) | workloadmeta collector "host" started successfully
2024-09-16 07:17:55 UTC | SECURITY | INFO | (comp/core/workloadmeta/store.go:540 in startCandidates) | workloadmeta collector "ecs_fargate" could not start. error: component workloadmeta-ecs_fargate is disabled: Agent is not running on ECS Fargate
2024-09-16 07:17:59 UTC | SECURITY | WARN | (subcommands/start/command.go:329 in StopAgent) | Some components were unhealthy: [workloadmeta-containerd workloadmeta-store workloadmeta-puller]
2024-09-16 07:17:59 UTC | SECURITY | INFO | (subcommands/start/command.go:346 in StopAgent) | See ya!
Stream closed EOF for kube-system/datadog-4mj27 (security-agent)

Describe what you expected:
Deployment successful without enabling the security-agent pod

Steps to reproduce the issue:
Deploy helm chart with defaults

Additional environment details (Operating System, Cloud provider, etc):
BottlerocketOS on EKS

In _helpers.tpl on line 353 it is not testing for true but instead testing for the key's existence?
https://github.com/DataDog/helm-charts/blob/main/charts/datadog/templates/_helpers.tpl#L353

[aquiladayc]

Hello @TheBlackMini, thanks for reaching out.

Curious about these messages. What Agent version do you use? I think it's introduced in 7.45, please upgrade your agent container image if you use older one.

2024-09-16 07:17:54 UTC | SECURITY | WARN | (pkg/config/model/viper.go:192 in checkKnownKey) | config key fips.enabled is unknown

I think we may need to see your setup, can you please reach out to our support.
https://www.datadoghq.com/support/

It says All security-agent components are deactivated, exiting, it's somehow enabled and maybe failed.
We would like to see

• values.yaml
• Debug flare, please see below steps
  • Set debug to datadog.logLevel (https://github.com/DataDog/helm-charts/blob/main/charts/datadog/values.yaml#L125)
  • Re-deploy the agent and send flare to us (https://docs.datadoghq.com/agent/troubleshooting/send_a_flare/?tab=agent#send-a-flare-using-the-flare-command)

By sending a flare, it will automatically create a support ticket for you.