Add a Kubernetes Distribution

[TylerHelmuth]

This PR adds a new distribution specifically for monitoring Kubernetes. For full details see #357.

Tested in https://github.com/TylerHelmuth/opentelemetry-collector-releases:

• action: https://github.com/TylerHelmuth/opentelemetry-collector-releases/actions/runs/8470931109
• release: https://github.com/TylerHelmuth/opentelemetry-collector-releases/releases/tag/v0.97.7
• packages:
  • https://github.com/TylerHelmuth/opentelemetry-collector-releases/pkgs/container/opentelemetry-collector-releases%2Fopentelemetry-collector-k8s
  • https://hub.docker.com/r/thelmuth34/opentelemetry-collector-k8s/tags

Example helm chart values.yaml using this distro:

mode: daemonset

clusterRole:
  create: true
  rules:
    - apiGroups: 
        - ""
      resources:
        - nodes/proxy
      verbs:
        - get

resources:
  limits:
    cpu: 200m
    memory: 512Mi
  requests:
    cpu: 100m
    memory: 256Mi

image:
  repository: thelmuth34/opentelemetry-collector-k8s
  tag: 0.97.7

command:
  name: otelcol-k8s

presets:
  logsCollection:
    enabled: true
  kubernetesAttributes:
    enabled: true
    extractAllPodLabels: true
    extractAllPodAnnotations: true
  kubeletMetrics:
    enabled: true

config:
  receivers:
    jaeger: null
    zipkin: null
    kubeletstats:
      collection_interval: 30s
      metric_groups:
        - node
        - pod
      metrics:
        k8s.node.uptime:
          enabled: true
        k8s.pod.uptime:
          enabled: true
        k8s.pod.cpu_limit_utilization:
          enabled: true
        k8s.pod.cpu_request_utilization:
          enabled: true
        k8s.pod.memory_limit_utilization:
          enabled: true
        k8s.pod.memory_request_utilization:
          enabled: true

  service:
    pipelines:
      traces:
        receivers: [otlp]
        exporters: [debug]
      metrics:
        exporters: [debug]
      logs:
        exporters: [debug]

ports:
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false

mode: deployment

image:
  repository: thelmuth34/opentelemetry-collector-k8s
  tag: 0.97.7

command:
  name: otelcol-k8s

resources:
  limits:
    cpu: 200m
    memory: 512Mi
  requests:
    cpu: 100m
    memory: 256Mi

# We only want one of these collectors - any more and we'd produce duplicate data
replicaCount: 1

presets:
  # enables the k8sclusterreceiver and adds it to the metrics pipelines
  clusterMetrics:
    enabled: true
  # enables the k8sobjectsreceiver to collect events only and adds it to the logs pipelines
  kubernetesEvents:
    enabled: true

config:
  receivers:
    k8s_cluster:
      collection_interval: 30s
    jaeger: null
    zipkin: null
  processors:
    transform/events:
      error_mode: ignore
      log_statements:
        - context: log
          statements:
            # adds a new watch-type attribute from the body if it exists
            - set(attributes["watch-type"], body["type"]) where IsMap(body) and body["type"] != nil

            # create new attributes from the body if the body is an object
            - merge_maps(attributes, body, "upsert") where IsMap(body) and body["object"] == nil
            - merge_maps(attributes, body["object"], "upsert") where IsMap(body) and body["object"] != nil

            # Transform the attributes so that the log events use the k8s.* semantic conventions
            - merge_maps(attributes, attributes[ "metadata"], "upsert") where IsMap(attributes[ "metadata"])
            - set(attributes["k8s.pod.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "Pod"
            - set(attributes["k8s.node.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "Node"
            - set(attributes["k8s.job.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "Job"
            - set(attributes["k8s.cronjob.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "CronJob"
            - set(attributes["k8s.namespace.name"], attributes["regarding"]["namespace"]) where attributes["regarding"]["kind"] == "Pod" or attributes["regarding"]["kind"] == "Job" or attributes["regarding"]["kind"] == "CronJob"

            # Transform the type attribtes into OpenTelemetry Severity types.
            - set(severity_text, attributes["type"]) where attributes["type"] == "Normal" or attributes["type"] == "Warning"
            - set(severity_number, SEVERITY_NUMBER_INFO) where attributes["type"] == "Normal"
            - set(severity_number, SEVERITY_NUMBER_WARN) where attributes["type"] == "Warning"


  service:
    pipelines:
      traces: null
      metrics:
        exporters: [ debug ]
      logs:
        processors: [ memory_limiter, transform/events, batch ]
        exporters: [ debug ]
        
ports:
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false

Closes #357

[TylerHelmuth]

Depends on open-telemetry/opentelemetry-helm-charts#1092

[dmitryax]

The set of components LGTM assuming we remove the deprecated logging exporter

[TylerHelmuth]

/cc @open-telemetry/operator-approvers

[jpkrohling]

It would be great to have the operator folks review this as well.

@avillela , you had a list of components during your talk yesterday and while I believe they are all here already, I'd appreciate it if you could double check.

[jpkrohling]

Thank you for making this work, @TylerHelmuth !

[swiatekm]

🚀

[avillela]

It would be great to have the operator folks review this as well.

@avillela , you had a list of components during your talk yesterday and while I believe they are all here already, I'd appreciate it if you could double check.

Are you referring to the Collector components for k8s @jpkrohling?

[mx-psi]

My general comment is "let's start smaller", by releasing for fewer OSs, architectures and types of artifacts

[TylerHelmuth]

@mx-psi @swiatekm-sumo you're comments on reducing the produced artifacts is valid, and annoying lol I will play around with what changes are necessary in cmd/goreleaser to reduce the artifact scope for this, and future, distros.

[TylerHelmuth]

@open-telemetry/helm-approvers please review

[dmitryax]

LGTM

[andrzej-stencel]

Shouldn't we add this distro to the main README.md file?

Current list of distributions:

• OpenTelemetry Collector (also known as "otelcol")
• OpenTelemetry Collector Contrib (also known as "otelcol-contrib")

[TylerHelmuth]

@astencel-sumo updated the readme and removed some statements that are no longer universally true.

[TylerHelmuth]

Ran another test against the latest commits in this branch (and 1 extra commit to update the references to my repo/dockerhub for the artifacts:

• https://github.com/TylerHelmuth/opentelemetry-collector-releases/actions/runs/8633831138
• https://github.com/TylerHelmuth/opentelemetry-collector-releases/actions/runs/8633831142
• https://github.com/TylerHelmuth/opentelemetry-collector-releases/actions/runs/8633831160

Artifacts: https://github.com/TylerHelmuth/opentelemetry-collector-releases/releases/tag/v0.97.9

Tested locally using the values.yamls from the description with the updated tag and can see the collector running as expected.

[jaronoff97]

🎉