Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

coredns无法解析,kube-proxy endpointslists请求数无法释放 #2167

Open
jingzhe6414 opened this issue Oct 12, 2024 · 1 comment
Open
Labels
kind/question kind/question

Comments

@jingzhe6414
Copy link

jingzhe6414 commented Oct 12, 2024

已经安装文档给的提示加入到了nodepool中

openyurt-helm版本1.5.0
raven kube-system 1 2024-10-11 22:42:14.090677672 +0800 CST deployed raven-agent-0.4.2 0.4.2 yurt-hub kube-system 1 2024-10-11 22:40:41.510098368 +0800 CST deployed yurthub-1.5.0 1.5.0 yurt-manager kube-system 1 2024-10-11 22:38:58.065591264 +0800 CST deployed yurt-manager-1.5.0 1.5.0

kubernetes version
root@sz-lt-i08-manager03:/home/megaium# kubectl version Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.2

查看endpointslists
root@server9:/home/megaium# cat /etc/kubernetes/cache/kube-proxy/endpointslices.v1.discovery.k8s.io/kube-system/kube-dns-vrr8q |jq . { "kind": "EndpointSlice", "apiVersion": "discovery.k8s.io/v1", "metadata": { "name": "kube-dns-vrr8q", "generateName": "kube-dns-", "namespace": "kube-system", "uid": "dbb73d26-f741-4e62-9ee7-0bf8a73a858a", "resourceVersion": "3209748", "generation": 404, "creationTimestamp": "2024-09-30T11:37:12Z", "labels": { "endpointslice.kubernetes.io/managed-by": "endpointslice-controller.k8s.io", "k8s-app": "kube-dns", "kubernetes.io/cluster-service": "true", "kubernetes.io/name": "KubeDNS", "kubernetes.io/service-name": "kube-dns" }, "annotations": { "endpoints.kubernetes.io/last-change-trigger-time": "2024-10-12T06:34:57Z" }, "ownerReferences": [ { "apiVersion": "v1", "kind": "Service", "name": "kube-dns", "uid": "02ca64b1-517a-4c3d-ac2a-6def0bb1cef0", "controller": true, "blockOwnerDeletion": true } ], "managedFields": [ { "manager": "kube-controller-manager", "operation": "Update", "apiVersion": "discovery.k8s.io/v1", "time": "2024-10-12T06:34:57Z", "fieldsType": "FieldsV1", "fieldsV1": { "f:addressType": {}, "f:endpoints": {}, "f:metadata": { "f:annotations": { ".": {}, "f:endpoints.kubernetes.io/last-change-trigger-time": {} }, "f:generateName": {}, "f:labels": { ".": {}, "f:endpointslice.kubernetes.io/managed-by": {}, "f:k8s-app": {}, "f:kubernetes.io/cluster-service": {}, "f:kubernetes.io/name": {}, "f:kubernetes.io/service-name": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"02ca64b1-517a-4c3d-ac2a-6def0bb1cef0\"}": {} } }, "f:ports": {} } } ] }, "addressType": "IPv4", "endpoints": [ { "addresses": [ "192.168.1.38" ], "conditions": { "ready": true, "serving": true, "terminating": false }, "targetRef": { "kind": "Pod", "namespace": "kube-system", "name": "coredns-n9xvn", "uid": "c692f328-41ab-4646-8ac8-8c92544f9ee1" }, "nodeName": "cx3001237923792379" } ], "ports": [ { "name": "metrics", "protocol": "TCP", "port": 9153 }, { "name": "dns", "protocol": "UDP", "port": 53 }, { "name": "dns-tcp", "protocol": "TCP", "port": 53 } ] }

node_yurthub_in_flight_requests_collector请求
`
root@server9:/home/megaium# curl http://127.0.0.1:10267/metrics |grep node_yurthub_proxy_traffic_collector
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 55027 0 55027 # HELP node_yurthub_proxy_traffic_collector collector of proxy response traffic by hub agent(unit: byte)
0 # TYPE node_yurthub_proxy_traffic_collector counter
0 36.node_yurthub_proxy_traffic_collector{client="coredns",resource="endpointslices",subresources="",verb="list"} 1.189629e+06
0M node_yurthub_proxy_traffic_collector{client="coredns",resource="endpointslices",subresources="",verb="watch"} 22954
node_yurthub_proxy_traffic_collector{client="coredns",resource="namespaces",subresources="",verb="list"} 201045
node_yurthub_proxy_traffic_collector{client="coredns",resource="namespaces",subresources="",verb="watch"} 1444
0node_yurthub_proxy_traffic_collector{client="coredns",resource="services",subresources="",verb="list"} 1.221222e+06
-node_yurthub_proxy_traffic_collector{client="coredns",resource="services",subresources="",verb="watch"} 4478
-:node_yurthub_proxy_traffic_collector{client="kube-proxy",resource="endpointslices",subresources="",verb="list"} 34933
--:node_yurthub_proxy_traffic_collector{client="kube-proxy",resource="endpointslices",subresources="",verb="watch"} 108596
-- node_yurthub_proxy_traffic_collector{client="kube-proxy",resource="events",subresources="",verb="create"} 991
-node_yurthub_proxy_traffic_collector{client="kube-proxy",resource="nodes",subresources="",verb="get"} 12859
-:-node_yurthub_proxy_traffic_collector{client="kube-proxy",resource="nodes",subresources="",verb="list"} 12903
-:-node_yurthub_proxy_traffic_collector{client="kube-proxy",resource="nodes",subresources="",verb="watch"} 8.555354e+06

  • node_yurthub_proxy_traffic_collector{client="kube-proxy",resource="services",subresources="",verb="list"} 31766
    -node_yurthub_proxy_traffic_collector{client="kube-proxy",resource="services",subresources="",verb="watch"} 11504
    -node_yurthub_proxy_traffic_collector{client="kubelet",resource="configmaps",subresources="",verb="list"} 37397
    :--:-- 52node_yurthub_proxy_traffic_collector{client="kubelet",resource="configmaps",subresources="",verb="watch"} 95736
    .4node_yurthub_proxy_traffic_collector{client="kubelet",resource="csidrivers",subresources="",verb="list"} 183
    M
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="csidrivers",subresources="",verb="watch"} 8439
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="csinodes",subresources="",verb="create"} 643
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="csinodes",subresources="",verb="get"} 1286
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="events",subresources="",verb="create"} 99383
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="events",subresources="",verb="patch"} 131354
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="nodes",subresources="",verb="create"} 4771
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="nodes",subresources="",verb="get"} 1.6368988e+07
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="nodes",subresources="",verb="list"} 7153
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="nodes",subresources="",verb="watch"} 8.609296e+06
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="nodes",subresources="status",verb="patch"} 287143
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="persistentvolumeclaims",subresources="",verb="get"} 326278
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="persistentvolumes",subresources="",verb="get"} 383630
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="pods",subresources="",verb="create"} 6082
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="pods",subresources="",verb="get"} 1.534219e+06
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="pods",subresources="",verb="list"} 55140
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="pods",subresources="",verb="watch"} 1.700328e+06
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="pods",subresources="status",verb="patch"} 1.434637e+06
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="runtimeclasses",subresources="",verb="list"} 183
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="runtimeclasses",subresources="",verb="watch"} 7395
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="secrets",subresources="",verb="list"} 1202
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="services",subresources="",verb="list"} 35935
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="services",subresources="",verb="watch"} 12516
    node_yurthub_proxy_traffic_collector{client="kubelet",resource="subjectaccessreviews",subresources="",verb="create"} 2571
    node_yurthub_proxy_traffic_collector{client="raven-agent-ds",resource="gateways",subresources="",verb="list"} 4717
    node_yurthub_proxy_traffic_collector{client="raven-agent-ds",resource="gateways",subresources="",verb="update"} 1425
    node_yurthub_proxy_traffic_collector{client="raven-agent-ds",resource="gateways",subresources="",verb="watch"} 16927
    node_yurthub_proxy_traffic_collector{client="yurthub",resource="configmaps",subresources="",verb="list"} 1196
    node_yurthub_proxy_traffic_collector{client="yurthub",resource="configmaps",subresources="",verb="watch"} 29480
    node_yurthub_proxy_traffic_collector{client="yurthub",resource="nodepools",subresources="",verb="list"} 2869
    node_yurthub_proxy_traffic_collector{client="yurthub",resource="nodepools",subresources="",verb="watch"} 12790
    node_yurthub_proxy_traffic_collector{client="yurthub",resource="nodes",subresources="",verb="get"} 204578
    node_yurthub_proxy_traffic_collector{client="yurthub",resource="services",subresources="",verb="list"} 23587
    node_yurthub_proxy_traffic_collector{client="yurthub",resource="services",subresources="",verb="watch"} 21781

`
多个边缘节点,目前只有该节点会出现这个问题,现在环境还存在可以提供

@jingzhe6414 jingzhe6414 added the kind/question kind/question label Oct 12, 2024
@luc99hen
Copy link
Member

Please post the log of yurthub.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/question kind/question
Projects
None yet
Development

No branches or pull requests

2 participants