Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

filter processor regex broken in 0.108.0 app version #36075

Open
ceastman-r7 opened this issue Oct 29, 2024 · 1 comment
Open

filter processor regex broken in 0.108.0 app version #36075

ceastman-r7 opened this issue Oct 29, 2024 · 1 comment
Labels
bug Something isn't working needs triage New item requiring triage processor/filter Filter processor

Comments

@ceastman-r7
Copy link

ceastman-r7 commented Oct 29, 2024

Component(s)

processor/filter

What happened?

Description

Going from app version 0.107.0 to 0.108.0 breaks the filter processor regex exclude.

Steps to Reproduce

using the following filter:
filter/dropmetrics:
error_mode: ignore
metrics:
exclude:
match_type: regexp
metric_names:
- containerd.*
- coredns.*
- datadog.*
- otel.*
- istio.mesh.*

Expected Result

I am expecting the metrics that start with these names will be filtered out (dropped).

Actual Result

metrics with these names are passed onto the exporters. i.e. otel.system.cpu.system

Collector version

0.107.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

exporters:
      datadog:
        api:
          key: ${env:DD_API_KEY}
          site: datadoghq.com
        metrics:
          resource_attributes_as_tags: true
      debug:
        verbosity: detailed
      otlphttp:
        auth:
          authenticator: basicauth/otlp
        endpoint: ${env:otel_otlp_endpoint}
      prometheusremotewrite/external:
        auth:
          authenticator: basicauth/metrics
        endpoint: ${env:otel_metrics_endpoint}
        resource_to_telemetry_conversion:
          enabled: true
        target_info:
          enabled: false
    extensions:
      basicauth/metrics:
        client_auth:
          password: ${env:otel_metrics_api}
          username: ${env:otel_metrics_user}
      basicauth/otlp:
        client_auth:
          password: ${env:otel_otlp_api}
          username: ${env:otel_otlp_user}
      file_storage:
        directory: /var/lib/otelcol
      health_check:
        endpoint: ${env:MY_POD_IP}:13133
    processors:
      attributes/addtags:
        actions:
        - action: insert
          key: kube_cluster_name
          value: ${env:K8S_CLUSTER_NAME}
        - action: insert
          key: otel_account_alias
          value: kubeplay-staging
      attributes/droptags:
        actions:
        - action: delete
          key: app_kubernetes_io_instance
        - action: delete
          key: app_kubernetes_io_name
        - action: delete
          key: cloud.platform
        - action: delete
          key: cloud.provider
        - action: delete
          key: cloud.region
        - action: delete
          key: cloud_platform
        - action: delete
          key: cloud_provider
        - action: delete
          key: cloud_region
        - action: delete
          key: container.id
        - action: delete
          key: container.image.name
        - action: delete
          key: container.runtime.version
        - action: delete
          key: container_id
        - action: delete
          key: container_image_name
        - action: delete
          key: container_runtime_version
        - action: delete
          key: destination_canonical_revision
        - action: delete
          key: destination_canonical_service
        - action: delete
          key: destination_cluster
        - action: delete
          key: family
        - action: delete
          key: host.id
        - action: delete
          key: host.image.id
        - action: delete
          key: host.name
        - action: delete
          key: host.type
        - action: delete
          key: host_id
        - action: delete
          key: host_image_id
        - action: delete
          key: host_name
        - action: delete
          key: host_type
        - action: delete
          key: http.scheme
        - action: delete
          key: http_scheme
        - action: delete
          key: instance
        - action: delete
          key: internal.ip
        - action: delete
          key: internal_ip
        - action: delete
          key: job
        - action: delete
          key: k8s.cronjob.uid
        - action: delete
          key: k8s.daemonset.uid
        - action: delete
          key: k8s.deployment.uid
        - action: delete
          key: k8s.hpa.uid
        - action: delete
          key: k8s.job.uid
        - action: delete
          key: k8s.namespace.uid
        - action: delete
          key: k8s.node.uid
        - action: delete
          key: k8s.pod.ip
        - action: delete
          key: k8s.pod.uid
        - action: delete
          key: k8s.replicaset.uid
        - action: delete
          key: k8s.statefulset.uid
        - action: delete
          key: k8s_cronjob_uid
        - action: delete
          key: k8s_daemonset_uid
        - action: delete
          key: k8s_deployment_uid
        - action: delete
          key: k8s_hpa_uid
        - action: delete
          key: k8s_job_uid
        - action: delete
          key: k8s_namespace_uid
        - action: delete
          key: k8s_node_uid
        - action: delete
          key: k8s_pod_ip
        - action: delete
          key: k8s_pod_uid
        - action: delete
          key: k8s_replicaset_uid
        - action: delete
          key: k8s_statefulset_uid
        - action: delete
          key: kubelet.version
        - action: delete
          key: kubelet_version
        - action: delete
          key: net.host.name
        - action: delete
          key: net.host.port
        - action: delete
          key: net_host_name
        - action: delete
          key: net_host_port
        - action: delete
          key: os.image
        - action: delete
          key: os.type
        - action: delete
          key: os_image
        - action: delete
          key: os_type
        - action: delete
          key: otel.scope.name
        - action: delete
          key: otel_scope_name
        - action: delete
          key: plugin
        - action: delete
          key: pod_template_hash
        - action: delete
          key: policy.background.mode
        - action: delete
          key: policy.type
        - action: delete
          key: policy_background_mode
        - action: delete
          key: policy_type
        - action: delete
          key: proto
        - action: delete
          key: provider.id
        - action: delete
          key: provider_id
        - action: delete
          key: request_protocol
        - action: delete
          key: resource.request.operation
        - action: delete
          key: resource_request_operation
        - action: delete
          key: response_flags
        - action: delete
          key: rule.execution.cause
        - action: delete
          key: rule.type
        - action: delete
          key: rule_execution_cause
        - action: delete
          key: rule_type
        - action: delete
          key: server
        - action: delete
          key: service.instance
        - action: delete
          key: service.instance.id
        - action: delete
          key: service_instance
        - action: delete
          key: service_instance_id
        - action: delete
          key: source_canonical_revision
        - action: delete
          key: source_canonical_service
        - action: delete
          key: source_cluster
        - action: delete
          key: source_principal
        - action: delete
          key: system.uuid
        - action: delete
          key: system_uuid
        - action: delete
          key: uid
        - action: delete
          key: zone
        - action: delete
          key: zones
      batch:
        send_batch_max_size: 100
        send_batch_size: 10
        timeout: 10s
      filter/dropmetrics:
        error_mode: ignore
        metrics:
          exclude:
            match_type: regexp
            metric_names:
            - containerd.*
            - coredns.*
            - datadog.*
            - otel.*
            - istio.mesh.*
      filter/ignorenamespaces:
        error_mode: ignore
        logs:
          log_record:
          - resource.attributes["namespace"] == "logging"
          - resource.attributes["namespace"] == "istio-addons"
          - resource.attributes["namespace"] == "istio-addons-ingress"
      filter/includemetrics:
        metrics:
          include:
            match_type: strict
            metric_names:
            - coredns_cache_hits_total
            - coredns_dns_request_duration_seconds
            - coredns_dns_requests_total
            - coredns_dns_responses_total
            - istio_tcp_connections_opened_total
            - k8s.container.cpu_request_utilization
            - k8s.container.cpu_request
            - k8s.container.memory_limit_utilization
            - k8s.container.memory_request_utilization
            - k8s.container.memory_request
            - k8s.container.ready
            - k8s.container.restarts
            - k8s.daemonset.desired_scheduled_nodes
            - k8s.daemonset.ready_nodes
            - k8s.deployment.available
            - k8s.deployment.desired
            - k8s.node.cpu.usage
            - k8s.node.memory.usage
            - k8s.node.network.errors
            - k8s.node.network.io
            - k8s.pod.cpu.usage
            - k8s.pod.memory.usage
            - k8s.pod.network.errors
            - k8s.pod.network.io
            - k8s.pod.phase
            - k8s.replicaset.available
            - k8s.replicaset.desired
            - k8s.statefulset.desired_pods
            - k8s.statefulset.ready_pods
            - kube_node_info
            - kube_node_spec_unschedulable
            - kube_node_status_allocatable
            - kube_node_status_capacity
            - kube_node_status_condition
            - kube_pod_container_info
            - kube_pod_container_resource_limits
            - kube_pod_container_resource_requests
            - kube_pod_container_status_ready
            - kube_pod_container_status_running
            - kube_pod_container_status_terminated
            - kube_pod_container_status_waiting_reason
            - kube_pod_container_status_waiting
            - kube_pod_info
            - kube_pod_init_container_info
            - kube_pod_init_container_status_ready
            - kube_pod_init_container_status_running
            - kube_pod_init_container_status_terminated
            - kube_pod_init_container_status_waiting
            - kube_service_info
            - policy_report_result
            - test.countermetric
            - test.gaugemetric
      filter/istiodinfo:
        error_mode: ignore
        logs:
          log_record:
          - IsMatch(body, ".*info.*") and resource.attributes["namespace"] == "istio-system"
            and resource.attributes["container"] == "discovery"
      k8sattributes:
        auth_type: serviceAccount
        extract:
          labels:
          - from: pod
            key: app.kubernetes.io/instance
            tag_name: kube_app_instance
          - from: pod
            key: app.kubernetes.io/version
            tag_name: kube_app_version
          - from: pod
            key: app.kubernetes.io/component
            tag_name: kube_app_component
          metadata:
          - container.image.tag
          - k8s.pod.name
          - k8s.pod.uid
          - k8s.deployment.name
          - k8s.node.name
          - k8s.namespace.name
          - k8s.daemonset.name
          - k8s.job.name
          - k8s.cronjob.name
          - k8s.statefulset.name
          - k8s.container.name
        passthrough: false
        pod_association:
        - sources:
          - from: resource_attribute
            name: k8s.pod.ip
        - sources:
          - from: resource_attribute
            name: k8s.pod.uid
      memory_limiter:
        check_interval: 5s
        limit_percentage: 80
        spike_limit_percentage: 25
      resource/droptags:
        attributes:
        - action: delete
          key: app_kubernetes_io_instance
        - action: delete
          key: app_kubernetes_io_name
        - action: delete
          key: cloud.platform
        - action: delete
          key: cloud.provider
        - action: delete
          key: cloud.region
        - action: delete
          key: cloud_platform
        - action: delete
          key: cloud_provider
        - action: delete
          key: cloud_region
        - action: delete
          key: container.id
        - action: delete
          key: container.image.name
        - action: delete
          key: container.runtime.version
        - action: delete
          key: container_id
        - action: delete
          key: container_image_name
        - action: delete
          key: container_runtime_version
        - action: delete
          key: destination_canonical_revision
        - action: delete
          key: destination_canonical_service
        - action: delete
          key: destination_cluster
        - action: delete
          key: family
        - action: delete
          key: host.id
        - action: delete
          key: host.image.id
        - action: delete
          key: host.name
        - action: delete
          key: host.type
        - action: delete
          key: host_id
        - action: delete
          key: host_image_id
        - action: delete
          key: host_name
        - action: delete
          key: host_type
        - action: delete
          key: http.scheme
        - action: delete
          key: http_scheme
        - action: delete
          key: instance
        - action: delete
          key: internal.ip
        - action: delete
          key: internal_ip
        - action: delete
          key: job
        - action: delete
          key: k8s.cronjob.uid
        - action: delete
          key: k8s.daemonset.uid
        - action: delete
          key: k8s.deployment.uid
        - action: delete
          key: k8s.hpa.uid
        - action: delete
          key: k8s.job.uid
        - action: delete
          key: k8s.namespace.uid
        - action: delete
          key: k8s.node.uid
        - action: delete
          key: k8s.pod.ip
        - action: delete
          key: k8s.pod.uid
        - action: delete
          key: k8s.replicaset.uid
        - action: delete
          key: k8s.statefulset.uid
        - action: delete
          key: k8s_cronjob_uid
        - action: delete
          key: k8s_daemonset_uid
        - action: delete
          key: k8s_deployment_uid
        - action: delete
          key: k8s_hpa_uid
        - action: delete
          key: k8s_job_uid
        - action: delete
          key: k8s_namespace_uid
        - action: delete
          key: k8s_node_uid
        - action: delete
          key: k8s_pod_ip
        - action: delete
          key: k8s_pod_uid
        - action: delete
          key: k8s_replicaset_uid
        - action: delete
          key: k8s_statefulset_uid
        - action: delete
          key: kubelet.version
        - action: delete
          key: kubelet_version
        - action: delete
          key: net.host.name
        - action: delete
          key: net.host.port
        - action: delete
          key: net_host_name
        - action: delete
          key: net_host_port
        - action: delete
          key: os.image
        - action: delete
          key: os.type
        - action: delete
          key: os_image
        - action: delete
          key: os_type
        - action: delete
          key: otel.scope.name
        - action: delete
          key: otel_scope_name
        - action: delete
          key: plugin
        - action: delete
          key: pod_template_hash
        - action: delete
          key: policy.background.mode
        - action: delete
          key: policy.type
        - action: delete
          key: policy_background_mode
        - action: delete
          key: policy_type
        - action: delete
          key: proto
        - action: delete
          key: provider.id
        - action: delete
          key: provider_id
        - action: delete
          key: request_protocol
        - action: delete
          key: resource.request.operation
        - action: delete
          key: resource_request_operation
        - action: delete
          key: response_flags
        - action: delete
          key: rule.execution.cause
        - action: delete
          key: rule.type
        - action: delete
          key: rule_execution_cause
        - action: delete
          key: rule_type
        - action: delete
          key: server
        - action: delete
          key: service.instance
        - action: delete
          key: service.instance.id
        - action: delete
          key: service_instance
        - action: delete
          key: service_instance_id
        - action: delete
          key: source_canonical_revision
        - action: delete
          key: source_canonical_service
        - action: delete
          key: source_cluster
        - action: delete
          key: source_principal
        - action: delete
          key: system.uuid
        - action: delete
          key: system_uuid
        - action: delete
          key: uid
        - action: delete
          key: zone
        - action: delete
          key: zones
      resource/loki:
        attributes:
        - action: upsert
          from_attribute: app.kubernetes.io/component
          key: component
        - action: insert
          from_attribute: k8s.container.name
          key: container
        - action: upsert
          from_attribute: release
          key: service.instance.id
        - action: upsert
          from_attribute: app.kubernetes.io/instance
          key: service.instance.id
        - action: insert
          from_attribute: k8s.namespace.name
          key: namespace
        - action: insert
          from_attribute: k8s.namespace.name
          key: service.namespace
        - action: insert
          from_attribute: app
          key: service.name
        - action: insert
          from_attribute: k8s.node.name
          key: node_name
        - action: insert
          from_attribute: k8s.pod.name
          key: pod
        - action: insert
          key: loki.resource.labels
          value: app, container, pod, namespace, node_name, response_code
        - action: insert
          key: loki.format
          value: raw
      resourcedetection:
        detectors:
        - env
        - eks
        - ec2
        - system
        override: false
        timeout: 2s
      transform/istioproxy:
        error_mode: ignore
        log_statements:
        - context: log
          statements:
          - merge_maps(cache, ParseJSON(body), "upsert") where resource.attributes["k8s.container.name"]
            == "istio-proxy" and IsMatch(body, "^\\{")
          - set(resource.attributes["authority"], cache["authority"])
          - set(resource.attributes["bytes_received"], cache["bytes_received"])
          - set(resource.attributes["connection_termination_details"], cache["connection_termination_details"])
          - set(resource.attributes["downstream_local_address"], cache["downstream_local_address"])
          - set(resource.attributes["downstream_remote_address"], cache["downstream_remote_address"])
          - set(resource.attributes["duration"], cache["duration"])
          - set(resource.attributes["method"], cache["method"])
          - set(resource.attributes["path"], cache["path"])
          - set(resource.attributes["protocol"], cache["protocol"])
          - set(resource.attributes["request_duration"], cache["request_duration"])
          - set(resource.attributes["request_id"], cache["request_id"])
          - set(resource.attributes["requested_server_name"], cache["requested_server_name"])
          - set(resource.attributes["response_code_details"], cache["auresponse_code_detailsthority"])
          - set(resource.attributes["response_code"], cache["response_code"])
          - set(resource.attributes["response_duration"], cache["response_duration"])
          - set(resource.attributes["response_flags"], cache["response_flags"])
          - set(resource.attributes["response_tx_duration"], cache["response_tx_duration"])
          - set(resource.attributes["start_time"], cache["start_time"])
          - set(resource.attributes["upstream_cluster"], cache["upstream_cluster"])
          - set(resource.attributes["upstream_host"], cache["upstream_host"])
          - set(resource.attributes["upstream_local_address"], cache["upstream_local_address"])
          - set(resource.attributes["upstream_transport_failure_reason"], cache["upstream_transport_failure_reason"])
          - set(resource.attributes["user_agent"], cache["user_agent"])
          - set(resource.attributes["x_b3_parentspanid"], cache["x_b3_parentspanid"])
          - set(resource.attributes["x_b3_traceid"], cache["x_b3_traceid"])
          - set(resource.attributes["x_forwarded_for"], cache["x_forwarded_for"])
      transform/log:
        log_statements:
        - context: log
          statements:
          - set(body, attributes["log"])
    receivers:
      datadog:
        endpoint: 0.0.0.0:8127
        read_timeout: 60s
      filelog:
        exclude:
        - /var/log/pods/platform-delivery_otel-ds*_*/opentelemetry-collector/*.log
        include:
        - /var/log/pods/*/*/*.log
        include_file_name: false
        include_file_path: true
        operators:
        - id: container-parser
          max_log_size: 102400
          type: container
        retry_on_failure:
          enabled: true
        start_at: end
        storage: file_storage
      jaeger:
        protocols:
          grpc:
            endpoint: ${env:MY_POD_IP}:14250
          thrift_compact:
            endpoint: ${env:MY_POD_IP}:6831
          thrift_http:
            endpoint: ${env:MY_POD_IP}:14268
      kubeletstats:
        auth_type: serviceAccount
        collection_interval: 60s
        endpoint: https://${env:KUBE_NODE_NAME}:10250
        extra_metadata_labels:
        - k8s.volume.type
        - container.id
        insecure_skip_verify: true
        metric_groups:
        - container
        - pod
        - volume
        - node
        metrics:
          container.cpu.usage:
            enabled: true
          container.uptime:
            enabled: true
          k8s.container.cpu.node.utilization:
            enabled: true
          k8s.container.cpu_limit_utilization:
            enabled: true
          k8s.container.cpu_request_utilization:
            enabled: true
          k8s.container.memory_limit_utilization:
            enabled: true
          k8s.container.memory_request_utilization:
            enabled: true
          k8s.node.cpu.usage:
            enabled: true
          k8s.node.uptime:
            enabled: true
          k8s.pod.cpu.node.utilization:
            enabled: true
          k8s.pod.cpu.usage:
            enabled: true
          k8s.pod.cpu_limit_utilization:
            enabled: true
          k8s.pod.cpu_request_utilization:
            enabled: true
          k8s.pod.memory_limit_utilization:
            enabled: true
          k8s.pod.memory_request_utilization:
            enabled: true
          k8s.pod.uptime:
            enabled: true
        node: ${env:KUBE_NODE_NAME}
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      prometheus:
        config:
          scrape_configs:
          - job_name: opentelemetry-collector
            scrape_interval: 10s
            static_configs:
            - targets:
              - ${env:MY_POD_IP}:8888
      zipkin:
        endpoint: ${env:MY_POD_IP}:9411
    service:
      extensions:
      - basicauth/metrics
      - basicauth/otlp
      - health_check
      - file_storage
      pipelines:
        logs:
          exporters:
          - otlphttp
          processors:
          - k8sattributes
          - resourcedetection
          - transform/istioproxy
          - resource/loki
          - transform/log
          - filter/istiodinfo
          - filter/ignorenamespaces
          receivers:
          - otlp
          - filelog
        metrics:
          exporters:
          - datadog
          processors:
          - attributes/addtags
          - resourcedetection
          - k8sattributes
          - filter/dropmetrics
          - resource/droptags
          - attributes/droptags
          - batch
          receivers:
          - otlp
          - datadog
        metrics/external:
          exporters:
          - prometheusremotewrite/external
          processors:
          - attributes/addtags
          - resourcedetection
          - k8sattributes
          - filter/includemetrics
          - resource/droptags
          - attributes/droptags
          - batch
          receivers:
          - kubeletstats
        metrics/externalotlp:
          exporters:
          - prometheusremotewrite/external
          processors:
          - attributes/addtags
          - resourcedetection
          - k8sattributes
          - filter/dropmetrics
          - resource/droptags
          - attributes/droptags
          - batch
          receivers:
          - otlp
          - datadog
        traces:
          exporters:
          - debug
          processors:
          - memory_limiter
          - batch
          receivers:
          - otlp
          - jaeger
          - zipkin
      telemetry:
        logs:
          encoding: json
        metrics:
          address: ${env:MY_POD_IP}:8888
          level: detailed

Log output

No response

Additional context

No response

@ceastman-r7 ceastman-r7 added bug Something isn't working needs triage New item requiring triage labels Oct 29, 2024
@github-actions github-actions bot added the processor/filter Filter processor label Oct 29, 2024
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@ceastman-r7 ceastman-r7 changed the title filter processor regex broken in 0.107.0 app version filter processor regex broken in 0.108.0 app version Oct 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs triage New item requiring triage processor/filter Filter processor
Projects
None yet
Development

No branches or pull requests

1 participant