Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prometheus-adapter-config only configure rules and externalRules, Crane will crash #829

Closed
aheizi opened this issue Jun 29, 2023 · 1 comment
Labels
kind/bug Something isn't working

Comments

@aheizi
Copy link

aheizi commented Jun 29, 2023

Describe the bug

When using Crane to dynamically load prometheus-adapter-config, if you only configure rules and externalRules, Crane will crash.

Configuration information:

rules:
- metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[30s])) by (<<.GroupBy>>)
  name:
    as: ${1}_qps
    matches: (.*)_total
  resources:
    overrides:
      namespace:
        resource: namespace
      pod:
        resource: pod
  seriesQuery: '{__name__=~"^http_requests.*_total$",container!="POD",namespace!="",pod!=""}'
externalRules:
- metricsQuery: avg(rate(<<.Series>>{<<.LabelMatchers>>}[30s])) by (name)
  resources:
    namespaced: false
  seriesQuery: 'http_requests_total'

Error message:

I0629 22:30:41.116099       1 config_fetcher.go:42] Got prometheus adapter configmap crane-system/prometheus-adapter
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x161a4d9]

goroutine 474 [running]:
github.com/gocrane/crane/pkg/prometheus-adapter.ParsingResourceRules(...)
    /go/src/github.com/gocrane/crane/pkg/prometheus-adapter/expression.go:65
github.com/gocrane/crane/pkg/prometheus-adapter.FlushRules({{0xc000cc8800, 0x1, 0x1}, 0x0, {0xc00000c960, 0x4, 0x4}}, {0x2361640, 0xc0000c6ff0})
    /go/src/github.com/gocrane/crane/pkg/prometheus-adapter/config_fetcher.go:138 +0xb9
github.com/gocrane/crane/pkg/prometheus-adapter.(*PrometheusAdapterConfigFetcher).Reconcile(0xc0007d4000, {0x2339878, 0xc000cdb8c0}, {{{0xc000430510, 0xc}, {0xc000b7c7e0, 0x12}}})
    /go/src/github.com/gocrane/crane/pkg/prometheus-adapter/config_fetcher.go:60 +0x525
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc0002246e0, {0x2339878, 0xc000cdb860}, {{{0xc000430510, 0x1e5b0a0}, {0xc000b7c7e0, 0xc0009c9a40}}})
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114 +0x222
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0002246e0, {0x23397d0, 0xc000922000}, {0x1dc7f80, 0xc000d0a600})
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311 +0x2f2
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0002246e0, {0x23397d0, 0xc000922000})
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:223 +0x354

Reproduce steps

  1. Prometheus adapter config
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-adapter
  namespace: crane-system
  labels:
    helm.sh/chart: prometheus-adapter-4.2.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: metrics
    app.kubernetes.io/part-of: prometheus-adapter
    app.kubernetes.io/name: prometheus-adapter
    app.kubernetes.io/instance: prometheus-adapter
    app.kubernetes.io/version: "v0.10.0"
data:
  config.yaml: |
    rules:
    - metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[30s])) by (<<.GroupBy>>)
      name:
        as: ${1}_qps
        matches: (.*)_total
      resources:
        overrides:
          namespace:
            resource: namespace
          pod:
            resource: pod
      seriesQuery: '{__name__=~"^http_requests.*_total$",container!="POD",namespace!="",pod!=""}'
    externalRules:
    - metricsQuery: avg(rate(<<.Series>>{<<.LabelMatchers>>}[30s])) by (name)
      resources:
        namespaced: false
      seriesQuery: 'http_requests_total'
  1. Crane start args
- --prometheus-adapter-configmap-namespace=crane-system
- --prometheus-adapter-configmap-name=prometheus-adapter
- --prometheus-adapter-configmap-key=config.yaml
- --prometheus-adapter-extension-labels=region="cn1"

Expected behavior

When only rules and externalRules are configured, metrics rules can be refreshed normally.

Screenshots
image

Environment (please complete the following information):

  • K8S Version: [e.g. 1.19]
  • Crane Version: [e.g. 0.1.0]
  • Browser [e.g. chrome, safari]
@aheizi aheizi added the kind/bug Something isn't working label Jun 29, 2023
@aheizi aheizi closed this as completed Jul 3, 2023
@aheizi aheizi reopened this Jul 3, 2023
@aheizi
Copy link
Author

aheizi commented Jul 3, 2023

this issue has already solved by #726

@aheizi aheizi closed this as completed Jul 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant