Lifecycle quit sometimes doesn't work

[Clusiv]

Hi. I faced strange behaviour when statsd_exporter receive post request on url /-/quit, print log
level=info ts=2022-08-10T12:57:38.099Z caller=main.go:543 msg="Received lifecycle api quit, exiting"
and nothing happen, container in pod still working.
We use it with airflow.
Full log:
level=info ts=2022-08-10T12:57:06.739Z caller=main.go:291 msg="Starting StatsD -> Prometheus Exporter" version="(version=0.22.4, branch=HEAD, revision=7e2fe6c2e61552062f03c6a5dddc219512456832)" level=info ts=2022-08-10T12:57:06.739Z caller=main.go:292 msg="Build context" context="(go=go1.17.3, user=root@f597e9e3e4dd, date=20211126-15:23:23)" level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.ti_failures level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.ti_successes level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.zombies_killed level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.scheduler_heartbeat level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.sla_email_notification_failure level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.dagbag_size level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.dag_processing.processes level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.scheduler.critical_section_busy level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.dag.callback_exceptions level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.celery.task_timeout_error level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.dag_processing.import_errors level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.dag_processing.total_parse_time level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.dag_processing.processor_timeouts level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.executor.open_slots level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.executor.queued_tasks level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.executor.running_tasks level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.smart_sensor_operator.poked_tasks level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.smart_sensor_operator.poked_success level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.smart_sensor_operator.poked_exception level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.smart_sensor_operator.exception_failures level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.smart_sensor_operator.infra_failures level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.scheduler.critical_section_duration level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.scheduler.tasks.killed_externally level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.scheduler.tasks.running level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.scheduler.tasks.starving level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.scheduler.orphaned_tasks.cleared level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.scheduler.orphaned_tasks.adopted level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.dag_processing.last_runtime.* level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.pool.open_slots.* level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.pool.queued_slots.* level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.pool.running_slots.* level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.pool.starving_tasks.* level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.dagrun.dependency-check.* level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.dag_processing.last_duration.* level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.dagrun.schedule_delay.* level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.dagrun.*.first_task_scheduling_delay level=warn ts=2022-08-10T12:57:06.743Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.ti.start.*.* level=warn ts=2022-08-10T12:57:06.744Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.dag_processing.last_run.seconds_ago.* level=warn ts=2022-08-10T12:57:06.744Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.dag.*.*.duration level=warn ts=2022-08-10T12:57:06.744Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.dagrun.duration.success.* level=warn ts=2022-08-10T12:57:06.744Z caller=fsm.go:314 msg="backtracking required because of match. Performance may be degraded" match=*.dagrun.duration.failed.* level=info ts=2022-08-10T12:57:06.744Z caller=main.go:341 msg="Accepting StatsD Traffic" udp=:8125 tcp=:9125 unixgram= level=info ts=2022-08-10T12:57:06.744Z caller=main.go:342 msg="Accepting Prometheus Requests" addr=:9102 level=debug ts=2022-08-10T12:57:30.249Z caller=listener.go:75 msg="Incoming line" proto=udp line=airflow-k8s.localtaskjob_start:1|c level=debug ts=2022-08-10T12:57:31.608Z caller=listener.go:75 msg="Incoming line" proto=udp line=airflow-k8s.localtaskjob_end:1|c level=info ts=2022-08-10T12:57:38.099Z caller=main.go:543 msg="Received lifecycle api quit, exiting

[matthiasr]

container in pod

I take it you are running the statsd exporter in Kubernetes? What is the restartPolicy? How many restarts does kubectl get pod show?

[matthiasr]

How many restarts does kubectl get pod show?

… and how does this number change when you call the quit endpoint?

[matthiasr]

Closing due to lack of feedback

[shreyash184]

@matthiasr can you help me I am facing the same issue.

StatsD logs -

[+] Running 1/0
✔ Container monitoring-dashboard-statsd-exporter-1 Created 0.0s
Attaching to statsd-exporter-1
statsd-exporter-1 | level=info ts=2024-04-08T11:56:09.511Z caller=main.go:321 msg="Starting StatsD -> Prometheus Exporter" version="(version=0.21.0, branch=HEAD, revision=ef6627b9f05350d54cd3bfea5afe36617d7eb5a4)"
statsd-exporter-1 | level=info ts=2024-04-08T11:56:09.513Z caller=main.go:322 msg="Build context" context="(go=go1.16.5, user=root@8ace135a0329, date=20210610-07:24:59)"
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.ti.start..*
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.dag_processing.last_run.seconds_ago.
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.dag...duration
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.dagrun.duration.success.*
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.dagrun.duration.failed.
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.ti_failures
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.ti_successes
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.zombies_killed
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.scheduler_heartbeat
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.sla_email_notification_failure
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.dagbag_size
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.dag_processing.processes
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.scheduler.critical_section_busy
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.dag.callback_exceptions
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.celery.task_timeout_error
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.dag_processing.import_errors
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.dag_processing.total_parse_time
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.dag_processing.processor_timeouts
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.executor.open_slots
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.executor.queued_tasks
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.executor.running_tasks
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.smart_sensor_operator.poked_tasks
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.smart_sensor_operator.poked_success
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.smart_sensor_operator.poked_exception
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.smart_sensor_operator.exception_failures
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.smart_sensor_operator.infra_failures
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.scheduler.critical_section_duration
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.scheduler.tasks.killed_externally
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.scheduler.tasks.running
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.scheduler.tasks.starving
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.scheduler.orphaned_tasks.cleared
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.scheduler.orphaned_tasks.adopted
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.dag_processing.last_runtime.*
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.pool.open_slots.
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.pool.queued_slots.
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.pool.running_slots.
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.pool.starving_tasks.
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.dagrun.dependency-check.
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.dag_processing.last_duration.
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.dagrun.schedule_delay.
statsd-exporter-1 | level=warn ts=2024-04-08T11:56:09.515Z caller=fsm.go:313 msg="backtracking required because of match. Performance may be degraded" match=.dagrun..first_task_scheduling_delay
statsd-exporter-1 | level=info ts=2024-04-08T11:56:09.515Z caller=main.go:361 msg="Accepting StatsD Traffic" udp=:9125 tcp=:9125 unixgram=
statsd-exporter-1 | level=info ts=2024-04-08T11:56:09.515Z caller=main.go:362 msg="Accepting Prometheus Requests" addr=:9102

you can see the code from here - https://github.com/shreyash184/Monitoring-Dashboard