Ingesters move to unhealthy state after startup

[sirishkumar]

I have deployed mimir-distributed with multitenancy disabled helm chart version 5.5.1 on kubernetes with IPV6 networking and istio service mesh enabled. Ingesters move to unhealthy state after some time after restart, I am unable to figure out why this is occuring.

Below are some warnings logs on restart of an ingester, does this indicate some problem?

mimir-ingester-2 ingester ts=2025-01-07T14:39:40.691751389Z caller=log.go:245 level=info msg="Suspect mimir-distributor-746cdcc45-kvrbm-2165d36d has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:39:45.69195705Z caller=log.go:245 level=info msg="Suspect mimir-store-gateway-0-f2ca5ce5 has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:39:50.692314509Z caller=log.go:245 level=info msg="Suspect mimir-compactor-0-53d5270a has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:39:55.692686956Z caller=log.go:245 level=info msg="Suspect mimir-ingester-0-d27c9a5b has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:40:00.693396933Z caller=log.go:245 level=info msg="Suspect mimir-ingester-1-742e3e66 has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:40:05.69380366Z caller=log.go:245 level=info msg="Suspect mimir-ruler-57f6dc4b6b-kgnbl-f054c843 has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:40:10.694606812Z caller=log.go:245 level=info msg="Suspect mimir-querier-689d9f6984-6vqsm-cc867397 has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:40:15.694936797Z caller=log.go:245 level=info msg="Suspect mimir-compactor-0-53d5270a has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:40:20.695267516Z caller=log.go:245 level=info msg="Suspect mimir-ingester-0-d27c9a5b has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:40:25.695941039Z caller=log.go:245 level=info msg="Suspect mimir-querier-689d9f6984-bmfgr-8e4df06c has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:40:35.696763885Z caller=log.go:245 level=info msg="Suspect mimir-ingester-1-742e3e66 has failed, no acks received"

Full logs

mimir-ingester-2 ingester ts=2025-01-07T14:39:30.691863943Z caller=memberlist_client.go:549 level=info msg="memberlist fast-join starting" nodes_found=9 to_join=4
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.692302176Z caller=log.go:245 level=debug msg="Initiating push/pull sync with:  [2a03:1e84:1902:11:8785::2]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.69404343Z caller=log.go:245 level=debug msg="Initiating push/pull sync with:  [2a03:1e84:1902:10:9d02::1]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.695788627Z caller=log.go:245 level=debug msg="Initiating push/pull sync with:  [2a03:1e84:1902:11:e1c4::3]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.697425994Z caller=log.go:245 level=debug msg="Initiating push/pull sync with:  [2a03:1e84:1902:12:f2d5::9]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.698779349Z caller=memberlist_client.go:569 level=info msg="memberlist fast-join finished" joined_nodes=4 elapsed_time=5.008054681s
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.698803126Z caller=memberlist_client.go:581 level=info phase=startup msg="joining memberlist cluster" join_members=dns+mimir-gossip-ring.bbobserve-metrics-bbobserve.svc.cluster.local:7946
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.699642262Z caller=log.go:245 level=debug msg="Initiating push/pull sync with:  [2a03:1e84:1902:11:e1c4::4]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.700647203Z caller=log.go:245 level=debug msg="Initiating push/pull sync with:  [2a03:1e84:1902:11:8785::2]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.70164125Z caller=log.go:245 level=debug msg="Initiating push/pull sync with:  [2a03:1e84:1902:11:e1c4::3]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.708583979Z caller=log.go:245 level=debug msg="Initiating push/pull sync with:  [2a03:1e84:1902:11:6ebf::e]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.70957397Z caller=log.go:245 level=debug msg="Initiating push/pull sync with:  [2a03:1e84:1902:11:b091::b]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.710582908Z caller=log.go:245 level=debug msg="Stream connection from=[2a03:1e84:1902:11:6ebf::7]:59672"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.710641835Z caller=log.go:245 level=debug msg="Initiating push/pull sync with:  [2a03:1e84:1902:11:6ebf::7]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.711888225Z caller=log.go:245 level=debug msg="Initiating push/pull sync with:  [2a03:1e84:1902:10:9d02::1]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.71383626Z caller=log.go:245 level=debug msg="Initiating push/pull sync with:  [2a03:1e84:1902:12:f2d5::d]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.7158898Z caller=log.go:245 level=debug msg="Initiating push/pull sync with:  [2a03:1e84:1902:12:f2d5::9]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.717128047Z caller=memberlist_client.go:588 level=info phase=startup msg="joining memberlist cluster succeeded" reached_nodes=9 elapsed_time=18.317969ms
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.821541291Z caller=sanity_check.go:44 level=info msg="Object storage config successfully checked"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.821575636Z caller=module_service.go:82 level=info msg=starting module=server
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.821637098Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=runtime-config waiting_for=server
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.821618574Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=memberlist-kv waiting_for=server
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.821650321Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=runtime-config waiting_for=activity-tracker
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.821651919Z caller=module_service.go:82 level=info msg=starting module=memberlist-kv
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.821658264Z caller=module_service.go:82 level=info msg=starting module=runtime-config
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.82167299Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=ingester-ring waiting_for=runtime-config
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.821819779Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=ingester-ring waiting_for=sanity-check
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.821828423Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=ingester-ring waiting_for=server
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.821832633Z caller=module_service.go:82 level=info msg=starting module=ingester-ring
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.822273262Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=ingester-service waiting_for=memberlist-kv
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.822280379Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=ingester-service waiting_for=runtime-config
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.822284373Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=ingester-service waiting_for=sanity-check
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.822288057Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=ingester-service waiting_for=server
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.822291848Z caller=module_service.go:82 level=info msg=starting module=ingester-service
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.822322269Z caller=ingester.go:2781 level=info msg="opening existing TSDBs"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.823368809Z caller=repair.go:56 level=info user=anonymous msg="Found healthy block" mint=1736239960000 maxt=1736244000000 ulid=01JH0B2HJHQRG25H0825B2HACR
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.824133238Z caller=repair.go:56 level=info user=anonymous msg="Found healthy block" mint=1736244000000 maxt=1736251200000 ulid=01JH0ECVT4GGDCZDVG2AP454Z5
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.838428868Z caller=head.go:658 level=info user=anonymous msg="Replaying on-disk memory mappable chunks if any"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.843568557Z caller=head.go:745 level=info user=anonymous msg="On-disk memory mappable chunks replay completed" duration=5.103024ms
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.843584951Z caller=head.go:753 level=info user=anonymous msg="Replaying WAL, this may take a while"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.858595852Z caller=head.go:790 level=info user=anonymous msg="WAL checkpoint loaded"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.858774008Z caller=head.go:825 level=info user=anonymous msg="WAL segment loaded" segment=11 maxSegment=18
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.860578949Z caller=head.go:825 level=info user=anonymous msg="WAL segment loaded" segment=12 maxSegment=18
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.860713635Z caller=head.go:825 level=info user=anonymous msg="WAL segment loaded" segment=13 maxSegment=18
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.868264736Z caller=head.go:825 level=info user=anonymous msg="WAL segment loaded" segment=14 maxSegment=18
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.8683856Z caller=head.go:825 level=info user=anonymous msg="WAL segment loaded" segment=15 maxSegment=18
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.868516572Z caller=head.go:825 level=info user=anonymous msg="WAL segment loaded" segment=16 maxSegment=18
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.87428292Z caller=head.go:825 level=info user=anonymous msg="WAL segment loaded" segment=17 maxSegment=18
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.874435328Z caller=head.go:825 level=info user=anonymous msg="WAL segment loaded" segment=18 maxSegment=18
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.874462144Z caller=head.go:862 level=info user=anonymous msg="WAL replay completed" checkpoint_replay_duration=15.030392ms wal_replay_duration=15.824603ms wbl_replay_duration=142ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=5.103024ms total_replay_duration=35.990066ms
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.876889428Z caller=db.go:2048 level=info user=anonymous msg="Compactions disabled"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.876900664Z caller=ingester.go:2694 level=info user=anonymous msg="Running compaction after WAL replay"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.877964905Z caller=ingester.go:2848 level=info msg="successfully opened existing TSDBs"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.878123171Z caller=mimir.go:890 level=info msg="Application started"
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.87899765Z caller=lifecycler.go:728 level=info msg="existing instance found in ring" state=ACTIVE tokens=512 ring=ingester readOnly=false readOnlyStateUpdate=0001-01-01T00:00:00Z
mimir-ingester-2 ingester ts=2025-01-07T14:39:30.879266214Z caller=lifecycler.go:540 level=debug msg="JoinAfter expired" ring=ingester
mimir-ingester-2 ingester ts=2025-01-07T14:39:33.888504526Z caller=ingester.go:3169 level=debug msg="TSDB blocks compaction completed successfully" user=anonymous compactReason=regular
mimir-ingester-2 ingester ts=2025-01-07T14:39:35.691740488Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-746cdcc45-kvrbm-2165d36d' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:37.692067273Z caller=log.go:245 level=debug msg="Failed UDP ping: mimir-distributor-746cdcc45-kvrbm-2165d36d (timeout reached)"
mimir-ingester-2 ingester ts=2025-01-07T14:39:37.692781825Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-746cdcc45-kvrbm-2165d36d' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:37.692964224Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-746cdcc45-kvrbm-2165d36d' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:37.692984018Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-746cdcc45-kvrbm-2165d36d' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:40.691751389Z caller=log.go:245 level=info msg="Suspect mimir-distributor-746cdcc45-kvrbm-2165d36d has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:39:40.692186813Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-store-gateway-0-f2ca5ce5' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:42.692932884Z caller=log.go:245 level=debug msg="Failed UDP ping: mimir-store-gateway-0-f2ca5ce5 (timeout reached)"
mimir-ingester-2 ingester ts=2025-01-07T14:39:42.693641264Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-store-gateway-0-f2ca5ce5' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:42.693870527Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-store-gateway-0-f2ca5ce5' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:42.693894033Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-store-gateway-0-f2ca5ce5' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:45.69195705Z caller=log.go:245 level=info msg="Suspect mimir-store-gateway-0-f2ca5ce5 has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:39:45.692403644Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-compactor-0-53d5270a' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:47.693232888Z caller=log.go:245 level=debug msg="Failed UDP ping: mimir-compactor-0-53d5270a (timeout reached)"
mimir-ingester-2 ingester ts=2025-01-07T14:39:47.693943003Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-compactor-0-53d5270a' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:47.694171252Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-compactor-0-53d5270a' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:47.694191119Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-compactor-0-53d5270a' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:50.692314509Z caller=log.go:245 level=info msg="Suspect mimir-compactor-0-53d5270a has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:39:50.692663313Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-ingester-0-d27c9a5b' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:52.693307882Z caller=log.go:245 level=debug msg="Failed UDP ping: mimir-ingester-0-d27c9a5b (timeout reached)"
mimir-ingester-2 ingester ts=2025-01-07T14:39:52.694216901Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-ingester-0-d27c9a5b' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:52.694241748Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-ingester-0-d27c9a5b' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:52.694260062Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-ingester-0-d27c9a5b' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:55.692686956Z caller=log.go:245 level=info msg="Suspect mimir-ingester-0-d27c9a5b has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:39:55.693035993Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-ingester-1-742e3e66' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:57.694020941Z caller=log.go:245 level=debug msg="Failed UDP ping: mimir-ingester-1-742e3e66 (timeout reached)"
mimir-ingester-2 ingester ts=2025-01-07T14:39:57.694878373Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-ingester-1-742e3e66' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:57.694904209Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-ingester-1-742e3e66' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:39:57.694917363Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-ingester-1-742e3e66' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:40:00.693396933Z caller=log.go:245 level=info msg="Suspect mimir-ingester-1-742e3e66 has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:40:00.693761001Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-ruler-57f6dc4b6b-kgnbl-f054c843' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:40:02.694657196Z caller=log.go:245 level=debug msg="Failed UDP ping: mimir-ruler-57f6dc4b6b-kgnbl-f054c843 (timeout reached)"
mimir-ingester-2 ingester ts=2025-01-07T14:40:02.695454352Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-ruler-57f6dc4b6b-kgnbl-f054c843' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:40:02.695480779Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-ruler-57f6dc4b6b-kgnbl-f054c843' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:40:05.69380366Z caller=log.go:245 level=info msg="Suspect mimir-ruler-57f6dc4b6b-kgnbl-f054c843 has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:40:05.694213747Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-querier-689d9f6984-6vqsm-cc867397' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:40:07.694948755Z caller=log.go:245 level=debug msg="Failed UDP ping: mimir-querier-689d9f6984-6vqsm-cc867397 (timeout reached)"
mimir-ingester-2 ingester ts=2025-01-07T14:40:07.695517522Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-querier-689d9f6984-6vqsm-cc867397' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:40:10.694606812Z caller=log.go:245 level=info msg="Suspect mimir-querier-689d9f6984-6vqsm-cc867397 has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:40:10.694975416Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-compactor-0-53d5270a' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:40:11.525605924Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: mimir-querier-689d9f6984-bmfgr-8e4df06c [::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:40:11.525751763Z caller=log.go:245 level=debug msg="Stream connection from=[::6]:58444"
mimir-ingester-2 ingester ts=2025-01-07T14:40:12.695282049Z caller=log.go:245 level=debug msg="Failed UDP ping: mimir-compactor-0-53d5270a (timeout reached)"
mimir-ingester-2 ingester ts=2025-01-07T14:40:12.695813014Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-compactor-0-53d5270a' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:40:15.694936797Z caller=log.go:245 level=info msg="Suspect mimir-compactor-0-53d5270a has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:40:15.695334556Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-ingester-0-d27c9a5b' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:40:17.695741099Z caller=log.go:245 level=debug msg="Failed UDP ping: mimir-ingester-0-d27c9a5b (timeout reached)"
mimir-ingester-2 ingester ts=2025-01-07T14:40:17.696309057Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-ingester-0-d27c9a5b' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:40:20.695267516Z caller=log.go:245 level=info msg="Suspect mimir-ingester-0-d27c9a5b has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:40:20.695612665Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-querier-689d9f6984-bmfgr-8e4df06c' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:40:22.695851842Z caller=log.go:245 level=debug msg="Failed UDP ping: mimir-querier-689d9f6984-bmfgr-8e4df06c (timeout reached)"
mimir-ingester-2 ingester ts=2025-01-07T14:40:25.695941039Z caller=log.go:245 level=info msg="Suspect mimir-querier-689d9f6984-bmfgr-8e4df06c has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:40:25.696327587Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-ingester-1-742e3e66' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:40:27.697335787Z caller=log.go:245 level=debug msg="Failed UDP ping: mimir-ingester-1-742e3e66 (timeout reached)"
mimir-ingester-2 ingester ts=2025-01-07T14:40:31.450226873Z caller=ingester.go:3038 level=debug msg="shipper successfully synchronized TSDB blocks with storage" user=anonymous uploaded=0
mimir-ingester-2 ingester ts=2025-01-07T14:40:32.745353646Z caller=logging.go:124 level=debug trace_id_unsampled=26c959f418bddcee sourceIPs=::6 msg="GET /ready (503) 97.578µs"
mimir-ingester-2 ingester ts=2025-01-07T14:40:35.696763885Z caller=log.go:245 level=info msg="Suspect mimir-ingester-1-742e3e66 has failed, no acks received"
mimir-ingester-2 ingester ts=2025-01-07T14:40:35.697152991Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-746cdcc45-kvrbm-2165d36d' from=[::6]:7946"
mimir-ingester-2 ingester ts=2025-01-07T14:40:37.697498855Z caller=log.go:245 level=debug msg="Failed UDP ping: mimir-distributor-746cdcc45-kvrbm-2165d36d (timeout reached)"

My deploy configuration

                values={
                    'alertmanager': {
                        'enabled': False,
                    },
                    'ingester': {
                        'zoneAwareReplication': {
                            'enabled': False,
                        },
                    },
                    'store_gateway': {
                        'zoneAwareReplication': {
                            'enabled': False,
                        },
                    },
                    'image': {
                        'repository': f'{metrics_deploy_config.comp_config.image_registry}/grafana/mimir',
                    },
                    'nginx': {
                        'image': {
                            'registry': metrics_deploy_config.comp_config.image_registry,
                        },
                    },
                    'rollout_operator': {
                        'image': {
                            'repository': f'{metrics_deploy_config.comp_config.image_registry}/grafana/rollout-operator',
                        }
                    },
                    'global': {
                        'podAnnotations': {
                            # Exclude the port used by the mimir for gossiping.
                            'traffic.sidecar.istio.io/excludeOutboundPorts': '7946,9095',
                            'traffic.sidecar.istio.io/excludeInboundPorts': '7946,9095',
                        },
                        'clusterDomain': 'cluster.local',  # Default values have a trailing dot leading to route resolution issues with istio.
                    },
                    'serviceAccount': {
                        'create': False,
                        'name': metrics_deploy_config.sa_mimir,
                    },
                    'minio': {
                        'enabled': False,
                    },
                    'rbac': {
                        'create': False,
                    },
                    'ruler': {
                        'enabled': True,
                        'serviceAccount': {
                            'create': False,
                            'name': metrics_deploy_config.sa_mimir,
                        },
                    },
                    'mimir': {
                        'structuredConfig': {
                            'multitenancy_enabled': False,
                            'usage_stats': {
                                'enabled': False,
                            },
                            'limits': {
                                'compactor_blocks_retention_period': f'{metrics_deploy_config.tenant_config.metrics.retention_period_days}d',  # type: ignore
                            },
                            'common': {
                                'storage': {
                                    'backend': 's3',
                                    's3': {
                                        'endpoint': f's3.{metrics_deploy_config.comp_config.aws_region}.amazonaws.com',
                                        'region': metrics_deploy_config.comp_config.aws_region,
                                    },
                                }
                            },
                            'blocks_storage': {
                                'backend': 's3',
                                's3': {
                                    'bucket_name': metrics_storage.block_storage.bucket.bucket,
                                },
                            },
                            'ruler_storage': {
                                'backend': 's3',
                                's3': {
                                    'bucket_name': metrics_storage.rules_storage.bucket.bucket,
                                },
                            },
                            'server': {
                                'log_level': 'debug',
                                'grpc_listen_address': '::0',
                                'http_listen_address': '::0',
                                'log_source_ips_enabled': True,
                            },
                            # ipv6 configuration
                            'memberlist': {
                                'cluster_label': 'mimir',
                                'cluster_label_verification_disabled': True,
                                'bind_addr': [
                                    '::',
                                ],
                            },
                            'ruler': {
                                'ring': {
                                    'instance_enable_ipv6': True,
                                }
                            },
                            'overrides_exporter': {
                                'ring': {
                                    'instance_enable_ipv6': True,
                                }
                            },
                            'frontend': {
                                'instance_enable_ipv6': True,
                            },
                            'distributor': {
                                'ring': {
                                    'instance_enable_ipv6': True,
                                },
                            },
                            'ingester': {
                                'ring': {
                                    'instance_enable_ipv6': True,
                                    'zone_awareness_enabled': False,
                                },
                            },
                            'query_scheduler': {
                                'ring': {
                                    'instance_enable_ipv6': True,
                                }
                            },
                            'store_gateway': {
                                'sharding_ring': {
                                    'instance_enable_ipv6': True,
                                    'zone_awareness_enabled': False,
                                }
                            },
                            'compactor': {
                                'sharding_ring': {
                                    'instance_enable_ipv6': True,
                                }
                            },
                        },
                    },
                    'continuous_test': {
                        'enabled': True,
                        'auth': {
                            'type': 'tenantId',
                            'tenant': 'anonymous',
                        },
                    },
                },