help request: etcd keeps failing

[arjunradiant]

Description

Hi I have installed etcd but only etcd-2 is up the others keep failing , how can I up the others aswell

Environment

• APISIX version (run apisix version):
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[wnz27]

use kubectl describe the pod and view the event‘s error？

[arjunradiant]

I was able to resolve it by deleting the pvc and creating and attaching a new pvc to it , i have one more doubt I have a service running in gke and for example the dns is www.xyz.com , i am using apisix gateway for it , my question is currently i am able to use only http://www.xyz.com but i want to use https , how can i do it , this is my ApisixRoute yaml

apiVersion: apisix.apache.org/v2beta3
kind: ApisixRoute
metadata:
name: xyz-route
spec:
http:

• name: xyz
  match:
  hosts:
  • xyz.com
    paths:
  • /xyz/*
    backends:
  • serviceName: xyz
    servicePort: 80
    plugins:
  • name: proxy-rewrite
    enable: true
    config:
    regex_uri:
    • ^/xyz/(.*)
    • "/$1"
  • name: cors
    enable: true
    config:
    allow_origins: ""
    allow_methods: "GET, POST, PUT, DELETE, PATCH, OPTIONS, MKCOL,COPY,MOVE,PROPFIND,PROPFIND,LOCK,UNLOCK,PATCH,TRACE"
    allow_headers: ""
    expose_headers: "*"
• name: swagger-ui
  match:
  hosts:
  • xyz.com
    paths:
  • /xyz/swagger-ui/*
    backends:
  • serviceName: xyz
    servicePort: 80
    plugins:
  • name: proxy-rewrite
    enable: true
    config:
    regex_uri:
    • ^/swagger-ui/(.*)
    • "/$1"
  • name: cors
    enable: true
    config:
    allow_origins: ""
    allow_methods: "GET, POST, PUT, DELETE, PATCH, OPTIONS, MKCOL,COPY,MOVE,PROPFIND,PROPFIND,LOCK,UNLOCK,PATCH,TRACE"
    allow_headers: ""
    expose_headers: "*"
• name: swagger-config
  match:
  hosts:
  • xyz.com
    paths:
  • /xyz/v3/*
    backends:
  • serviceName: xyz
    servicePort: 80
    plugins:
  • name: proxy-rewrite
    enable: true
    config:
    regex_uri:
    • ^(xyz/v3/.*)
    • "/$1"
  • name: cors
    enable: true
    config:
    allow_origins: ""
    allow_methods: "GET, POST, PUT, DELETE, PATCH, OPTIONS, MKCOL,COPY,MOVE,PROPFIND,PROPFIND,LOCK,UNLOCK,PATCH,TRACE"
    allow_headers: ""
    expose_headers: "*"
• name: feeservice-wildcard
  match:
  hosts:
  • xyz.com
    paths:
  • "/-xyz/"
    backends:
  • serviceName: xyz
    servicePort: 80
    plugins:
  • name: proxy-rewrite
    enable: true
    config:
    regex_uri:
    • ^/(.*)
    • "/$1"
  • name: cors
    enable: true
    config:
    allow_origins: ""
    allow_methods: "GET, POST, PUT, DELETE, PATCH, OPTIONS, MKCOL,COPY,MOVE,PROPFIND,PROPFIND,LOCK,UNLOCK,PATCH,TRACE"
    allow_headers: ""
    expose_headers: "*"

[arjunradiant]

please can you help @wnz27 @indrekj @huacnlee @markokocic

[arjunradiant]

please can someone help me regarding this_

I was able to resolve it by deleting the pvc and creating and attaching a new pvc to it , i have one more doubt I have a service running in gke and for example the dns is www.xyz.com , i am using apisix gateway for it , my question is currently i am able to use only http://www.xyz.com/ but i want to use https , how can i do it , this is my ApisixRoute yaml

apiVersion: apisix.apache.org/v2beta3
kind: ApisixRoute
metadata:
name: xyz-route
spec:
http:

name: xyz
match:
hosts:
xyz.com
paths:
/xyz/*
backends:
serviceName: xyz
servicePort: 80
plugins:
name: proxy-rewrite
enable: true
config:
regex_uri:
^/xyz/(.)
"/$1"
name: cors
enable: true
config:
allow_origins: ""
allow_methods: "GET, POST, PUT, DELETE, PATCH, OPTIONS, MKCOL,COPY,MOVE,PROPFIND,PROPFIND,LOCK,UNLOCK,PATCH,TRACE"
allow_headers: ""
expose_headers: ""
name: swagger-ui
match:
hosts:
xyz.com
paths:
/xyz/swagger-ui/*
backends:
serviceName: xyz
servicePort: 80
plugins:
name: proxy-rewrite
enable: true
config:
regex_uri:
^/swagger-ui/(.)
"/$1"
name: cors
enable: true
config:
allow_origins: ""
allow_methods: "GET, POST, PUT, DELETE, PATCH, OPTIONS, MKCOL,COPY,MOVE,PROPFIND,PROPFIND,LOCK,UNLOCK,PATCH,TRACE"
allow_headers: ""
expose_headers: ""
name: swagger-config
match:
hosts:
xyz.com
paths:
/xyz/v3/*
backends:
serviceName: xyz
servicePort: 80
plugins:
name: proxy-rewrite
enable: true
config:
regex_uri:
^(xyz/v3/.)
"/$1"
name: cors
enable: true
config:
allow_origins: ""
allow_methods: "GET, POST, PUT, DELETE, PATCH, OPTIONS, MKCOL,COPY,MOVE,PROPFIND,PROPFIND,LOCK,UNLOCK,PATCH,TRACE"
allow_headers: ""
expose_headers: ""
name: feeservice-wildcard
match:
hosts:
xyz.com
paths:
"/-xyz/"
backends:
serviceName: xyz
servicePort: 80
plugins:
name: proxy-rewrite
enable: true
config:
regex_uri:
^/(.)
"/$1"
name: cors
enable: true
config:
allow_origins: ""
allow_methods: "GET, POST, PUT, DELETE, PATCH, OPTIONS, MKCOL,COPY,MOVE,PROPFIND,PROPFIND,LOCK,UNLOCK,PATCH,TRACE"
allow_headers: ""
expose_headers: ""

[wnz27]

My point is that the etcd error has nothing to do with the apisixroute resource definition.

[arjunradiant]

the https issue is resolved but my question still is why does my etcd keep going down @wnz27 , also the false alarm rings in etcd when usage reaches 30 %

[Baoyuantop]

Hi @arjunradiant, any error logs?

[arjunradiant]

@Baoyuantop @wnz27 i am sharing etcd logs

{"level":"info","ts":"2025-01-13T12:56:14.447987Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d [logterm: 63, index: 2008533] sent MsgPreVote request to 3e8d9a4c0aedfad at term 63"}
{"level":"warn","ts":"2025-01-13T12:56:15.919735Z","caller":"etcdserver/server.go:2083","msg":"failed to publish local member to cluster through raft","local-member-id":"39e3f227d2d33a6d","local-member-attributes":"{Name:apisix-etcd-2 ClientURLs:[http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379 http://apisix-etcd.ingress-apisix.svc.cluster.local:2379]}","request-path":"/0/members/39e3f227d2d33a6d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2025-01-13T12:56:15.947285Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d is starting a new election at term 63"}
{"level":"info","ts":"2025-01-13T12:56:15.947362Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d became pre-candidate at term 63"}
{"level":"info","ts":"2025-01-13T12:56:15.947382Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d received MsgPreVoteResp from 39e3f227d2d33a6d at term 63"}
{"level":"info","ts":"2025-01-13T12:56:15.947441Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d [logterm: 63, index: 2008533] sent MsgPreVote request to 3e8d9a4c0aedfad at term 63"}
{"level":"warn","ts":"2025-01-13T12:56:16.711767Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"3e8d9a4c0aedfad","rtt":"2.995567009s","error":"dial tcp 10.48.1.212:2380: connect: connection refused"}
{"level":"warn","ts":"2025-01-13T12:56:16.79283Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"3e8d9a4c0aedfad","rtt":"2.997341995s","error":"dial tcp 10.48.1.212:2380: connect: connection refused"}
{"level":"info","ts":"2025-01-13T12:56:17.44753Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d is starting a new election at term 63"}
{"level":"info","ts":"2025-01-13T12:56:17.447626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d became pre-candidate at term 63"}
{"level":"info","ts":"2025-01-13T12:56:17.447658Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d received MsgPreVoteResp from 39e3f227d2d33a6d at term 63"}
{"level":"info","ts":"2025-01-13T12:56:17.447685Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d [logterm: 63, index: 2008533] sent MsgPreVote request to 3e8d9a4c0aedfad at term 63"}
{"level":"info","ts":"2025-01-13T12:56:18.947182Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d is starting a new election at term 63"}
{"level":"info","ts":"2025-01-13T12:56:18.947248Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d became pre-candidate at term 63"}
{"level":"info","ts":"2025-01-13T12:56:18.947271Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d received MsgPreVoteResp from 39e3f227d2d33a6d at term 63"}
{"level":"info","ts":"2025-01-13T12:56:18.947296Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d [logterm: 63, index: 2008533] sent MsgPreVote request to 3e8d9a4c0aedfad at term 63"}
{"level":"info","ts":"2025-01-13T12:56:20.447656Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d is starting a new election at term 63"}
{"level":"info","ts":"2025-01-13T12:56:20.447752Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d became pre-candidate at term 63"}
{"level":"info","ts":"2025-01-13T12:56:20.447775Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d received MsgPreVoteResp from 39e3f227d2d33a6d at term 63"}
{"level":"info","ts":"2025-01-13T12:56:20.447802Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d [logterm: 63, index: 2008533] sent MsgPreVote request to 3e8d9a4c0aedfad at term 63"}
{"level":"warn","ts":"2025-01-13T12:56:21.712686Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"3e8d9a4c0aedfad","rtt":"2.995567009s","error":"dial tcp 10.48.1.212:2380: connect: connection refused"}
{"level":"warn","ts":"2025-01-13T12:56:21.793476Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"3e8d9a4c0aedfad","rtt":"2.997341995s","error":"dial tcp 10.48.1.212:2380: connect: connection refused"}
{"level":"info","ts":"2025-01-13T12:56:21.947891Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d is starting a new election at term 63"}
{"level":"info","ts":"2025-01-13T12:56:21.94796Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d became pre-candidate at term 63"}
{"level":"info","ts":"2025-01-13T12:56:21.947981Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d received MsgPreVoteResp from 39e3f227d2d33a6d at term 63"}
{"level":"info","ts":"2025-01-13T12:56:21.948019Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d [logterm: 63, index: 2008533] sent MsgPreVote request to 3e8d9a4c0aedfad at term 63"}
{"level":"warn","ts":"2025-01-13T12:56:22.920145Z","caller":"etcdserver/server.go:2083","msg":"failed to publish local member to cluster through raft","local-member-id":"39e3f227d2d33a6d","local-member-attributes":"{Name:apisix-etcd-2 ClientURLs:[http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379 http://apisix-etcd.ingress-apisix.svc.cluster.local:2379]}","request-path":"/0/members/39e3f227d2d33a6d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2025-01-13T12:56:23.447518Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d is starting a new election at term 63"}
{"level":"info","ts":"2025-01-13T12:56:23.447598Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d became pre-candidate at term 63"}
{"level":"info","ts":"2025-01-13T12:56:23.447621Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d received MsgPreVoteResp from 39e3f227d2d33a6d at term 63"}
{"level":"info","ts":"2025-01-13T12:56:23.447646Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d [logterm: 63, index: 2008533] sent MsgPreVote request to 3e8d9a4c0aedfad at term 63"}
{"level":"info","ts":"2025-01-13T12:56:24.946915Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d is starting a new election at term 63"}
{"level":"info","ts":"2025-01-13T12:56:24.947109Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d became pre-candidate at term 63"}
{"level":"info","ts":"2025-01-13T12:56:24.947134Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d received MsgPreVoteResp from 39e3f227d2d33a6d at term 63"}
{"level":"info","ts":"2025-01-13T12:56:24.947438Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d [logterm: 63, index: 2008533] sent MsgPreVote request to 3e8d9a4c0aedfad at term 63"}
{"level":"info","ts":"2025-01-13T12:56:26.447864Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d is starting a new election at term 63"}
{"level":"info","ts":"2025-01-13T12:56:26.447964Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d became pre-candidate at term 63"}
{"level":"info","ts":"2025-01-13T12:56:26.447985Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d received MsgPreVoteResp from 39e3f227d2d33a6d at term 63"}
{"level":"info","ts":"2025-01-13T12:56:26.448041Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d [logterm: 63, index: 2008533] sent MsgPreVote request to 3e8d9a4c0aedfad at term 63"}
{"level":"warn","ts":"2025-01-13T12:56:26.713795Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"3e8d9a4c0aedfad","rtt":"2.995567009s","error":"dial tcp 10.48.1.212:2380: connect: connection refused"}
{"level":"warn","ts":"2025-01-13T12:56:26.794646Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"3e8d9a4c0aedfad","rtt":"2.997341995s","error":"dial tcp 10.48.1.212:2380: connect: connection refused"}
{"level":"info","ts":"2025-01-13T12:56:27.94691Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d is starting a new election at term 63"}
{"level":"info","ts":"2025-01-13T12:56:27.94698Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d became pre-candidate at term 63"}
{"level":"info","ts":"2025-01-13T12:56:27.947017Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d received MsgPreVoteResp from 39e3f227d2d33a6d at term 63"}
{"level":"info","ts":"2025-01-13T12:56:27.947043Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d [logterm: 63, index: 2008533] sent MsgPreVote request to 3e8d9a4c0aedfad at term 63"}
{"level":"info","ts":"2025-01-13T12:56:29.446878Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d is starting a new election at term 63"}
{"level":"info","ts":"2025-01-13T12:56:29.446974Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d became pre-candidate at term 63"}
{"level":"info","ts":"2025-01-13T12:56:29.446997Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d received MsgPreVoteResp from 39e3f227d2d33a6d at term 63"}
{"level":"info","ts":"2025-01-13T12:56:29.447294Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d [logterm: 63, index: 2008533] sent MsgPreVote request to 3e8d9a4c0aedfad at term 63"}
{"level":"warn","ts":"2025-01-13T12:56:29.920893Z","caller":"etcdserver/server.go:2083","msg":"failed to publish local member to cluster through raft","local-member-id":"39e3f227d2d33a6d","local-member-attributes":"{Name:apisix-etcd-2 ClientURLs:[http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379 http://apisix-etcd.ingress-apisix.svc.cluster.local:2379]}","request-path":"/0/members/39e3f227d2d33a6d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2025-01-13T12:56:30.947233Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d is starting a new election at term 63"}
{"level":"info","ts":"2025-01-13T12:56:30.947319Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"39e3f227d2d33a6d became pre-candidate at term 63"}

[arjunradiant]

this is my apisix logs

025/01/13 13:04:31 [error] 51#51: *13355164 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2025/01/13 13:04:31 [error] 54#54: *13355165 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2025/01/13 13:04:31 [error] 50#50: *13355166 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2025/01/13 13:04:31 [warn] 54#54: *13354830 [lua] v3.lua:247: _request_uri(): http://apisix-etcd.ingress-apisix.svc.cluster.local:2379: timeout. Retrying, context: ngx.timer
2025/01/13 13:04:31 [error] 54#54: *13354830 [lua] server-info.lua:187: report(): failed to get server_info from etcd: has no healthy etcd endpoint available, context: ngx.timer
2025/01/13 13:04:32 [error] 54#54: *13355178 [lua] server-info.lua:187: report(): failed to get server_info from etcd: has no healthy etcd endpoint available, context: ngx.timer
2025/01/13 13:04:33 [error] 51#51: *13354975 [lua] resource.lua:169: failed to get upstream[/upstreams/29b61b45] from etcd: has no healthy etcd endpoint available, client: 10.48.6.82, server: , request: "GET /apisix/admin/upstreams/29b61b45 HTTP/1.1", host: "apisix-admin.ingress-apisix.svc.cluster.local:9180"
2025/01/13 13:04:33 [warn] 51#51: *13354860 [lua] v3.lua:247: _request_uri(): http://apisix-etcd.ingress-apisix.svc.cluster.local:2379: timeout. Retrying, client: 10.48.6.82, server: , request: "GET /apisix/admin/upstreams/29b61b45 HTTP/1.1", host: "apisix-admin.ingress-apisix.svc.cluster.local:9180"
2025/01/13 13:04:33 [error] 51#51: *13354860 [lua] resource.lua:169: failed to get upstream[/upstreams/29b61b45] from etcd: has no healthy etcd endpoint available, client: 10.48.6.82, server: , request: "GET /apisix/admin/upstreams/29b61b45 HTTP/1.1", host: "apisix-admin.ingress-apisix.svc.cluster.local:9180"
2025/01/13 13:04:34 [error] 51#51: *13355195 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2025/01/13 13:04:34 [error] 54#54: *13355196 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2025/01/13 13:04:34 [error] 50#50: *13355197 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2025/01/13 13:04:35 [warn] 51#51: *13354880 [lua] v3.lua:247: _request_uri(): http://apisix-etcd.ingress-apisix.svc.cluster.local:2379: timeout. Retrying, client: 10.48.6.82, server: , request: "PUT /apisix/admin/routes/38762963 HTTP/1.1", host: "apisix-admin.ingress-apisix.svc.cluster.local:9180"
2025/01/13 13:04:35 [warn] 51#51: *13354882 [lua] v3.lua:247: _request_uri(): http://apisix-etcd.ingress-apisix.svc.cluster.local:2379: timeout. Retrying, client: 10.48.6.82, server: , request: "GET /apisix/admin/ssls/51375331 HTTP/1.1", host: "apisix-admin.ingress-apisix.svc.cluster.local:9180"
2025/01/13 13:04:35 [error] 51#51: *13354882 [lua] resource.lua:169: failed to get ssl[/ssls/51375331] from etcd: has no healthy etcd endpoint available, client: 10.48.6.82, server: , request: "GET /apisix/admin/ssls/51375331 HTTP/1.1", host: "apisix-admin.ingress-apisix.svc.cluster.local:9180"
10.48.6.82 - - [13/Jan/2025:13:04:33 +0000] apisix-admin.ingress-apisix.svc.cluster.local:9180 "GET /apisix/admin/upstreams/29b61b45 HTTP/1.1" 503 66 0.000 "-" "Go-http-client/1.1" - - - "http://apisix-admin.ingress-apisix.svc.cluster.local:9180"
10.48.6.82 - - [13/Jan/2025:13:04:33 +0000] apisix-admin.ingress-apisix.svc.cluster.local:9180 "GET /apisix/admin/upstreams/29b61b45 HTTP/1.1" 503 66 30.001 "-" "Go-http-client/1.1" - - - "http://apisix-admin.ingress-apisix.svc.cluster.local:9180"
10.48.6.82 - - [13/Jan/2025:13:04:35 +0000] apisix-admin.ingress-apisix.svc.cluster.local:9180 "PUT /apisix/admin/routes/38762963 HTTP/1.1" 400 123 29.999 "-" "Go-http-client/1.1" - - - "http://apisix-admin.ingress-apisix.svc.cluster.local:9180"
10.48.6.82 - - [13/Jan/2025:13:04:35 +0000] apisix-admin.ingress-apisix.svc.cluster.local:9180 "GET /apisix/admin/ssls/51375331 HTTP/1.1" 503 66 30.000 "-" "Go-http-client/1.1" - - - "http://apisix-admin.ingress-apisix.svc.cluster.local:9180"
2025/01/13 13:04:37 [error] 51#51: *13355226 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2025/01/13 13:04:37 [error] 54#54: *13355227 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2025/01/13 13:04:37 [error] 50#50: *13355228 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer

[wnz27]

apisix check the etcd healthz not through.
and etcd errorlog is "failed to publish local member to cluster through raft "
proberbly is web problem ? when join node to etcd error happen.

[juzhiyuan]

I noticed you are using etcd inside K8s as Pods. It's a known issue in etcd community and as far as I know, the only one stable solution is to deploy ETCD on a VM. Correct me if I'm wrong

[arjunradiant]

@juzhiyuan @wnz27 , the reason i deployed it on k8s is because its being connected to k8s services for the routes , if i deploy it in a vm can i connect the apisix to the k8s services and create routes ? and will the etcd issue be reolved because it keeps failing after sometime , I was in a thought that this may be due to the cpu and ram utilization in the etcd pods

[wnz27]

@juzhiyuan @wnz27 , the reason i deployed it on k8s is because its being connected to k8s services for the routes , if i deploy it in a vm can i connect the apisix to the k8s services and create routes ? and will the etcd issue be reolved because it keeps failing after sometime , I was in a thought that this may be due to the cpu and ram utilization in the etcd pods

if etcd deploy on vm, first u guarantee the web and host can access, and then u run apisix must change the helm template's config.

[juzhiyuan]

@juzhiyuan @wnz27 , the reason i deployed it on k8s is because its being connected to k8s services for the routes , if i deploy it in a vm can i connect the apisix to the k8s services and create routes ? and will the etcd issue be reolved because it keeps failing after sometime , I was in a thought that this may be due to the cpu and ram utilization in the etcd pods

Yes. Deploying ETCD outside of K8s (e.g., on a VM) is much more stable than your current usage, even though it sounds "weird"

I know a few open-source users who are using etcd within Kubernetes. Whenever the etcd pods crash, they run kubectl apply to update the configurations again, because etcd is a configuration store, it won't affect the gateway's stability and performance, as far as I know.

Another piece of information I can share is that since API7 Enterprise uses PostgreSQL instead of ETCD, the API7 Ingress Controller will not face the issues currently experienced by APISIX, and we plan to release the GA version in mid-May.