Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deploy craned didn't READY #763

Closed
flying1574 opened this issue May 6, 2023 · 6 comments
Closed

deploy craned didn't READY #763

flying1574 opened this issue May 6, 2023 · 6 comments
Labels
kind/bug Something isn't working

Comments

@flying1574
Copy link

Describe the bug
deploy craned didn't ready
Reproduce steps
follow the install steps,and Verify Installation ,get the status

Expected behavior
like the installation guide,the deploy craned should be Ready and available
Screenshots
image

Environment (please complete the following information):

  • K8S Version: [e.g. v1.27.1]
  • Crane Version: [e.g. 0.1.0]
  • Browser [e.g. chrome, safari]
@flying1574 flying1574 added the kind/bug Something isn't working label May 6, 2023
@qmhu
Copy link
Member

qmhu commented May 7, 2023

Can't see any error logs, pls provide more info.

@flying1574
Copy link
Author

my local kubernetes cluster based on centos 7.9,the kubernetes version is v1.27.1, hem version is v3.11.3, kind version is 0.18.0,
when I verify installation use kubectl get deploy -n crane-system， I found the deploy "craned" didn't ready, and I also to check the logs, no any error

@flying1574
Copy link
Author

image

[root@k8s-master01 crane-lab]# kubectl describe deploy/craned -n crane-system
Name: craned
Namespace: crane-system
CreationTimestamp: Sun, 07 May 2023 11:34:40 +0800
Labels: app=craned
app.kubernetes.io/instance=crane
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=crane
app.kubernetes.io/version=v0.10.0
helm.sh/chart=crane-0.10.0
Annotations: deployment.kubernetes.io/revision: 1
meta.helm.sh/release-name: crane
meta.helm.sh/release-namespace: crane-system
Selector: app=craned
Replicas: 1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=craned
Service Account: craned
Containers:
craned:
Image: docker.io/gocrane/craned:v0.10.0
Port: 9443/TCP
Host Port: 0/TCP
Command:
/craned
Args:
--recommendation-configuration-file=/tmp/recommendation-config/config.yaml
--feature-gates=Analysis=true,TimeSeriesPrediction=true,Autoscaling=true,NodeResource=false
--prometheus-address=http://prometheus-server.crane-system.svc.cluster.local:8080
--v=2
Readiness: http-get http://:8082/api/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
TZ: Asia/Shanghai
CRANE_SYSTEM_NAMESPACE: (v1:metadata.namespace)
Mounts:
/tmp/k8s-webhook-server/serving-certs from cert (ro)
/tmp/recommendation-config from config (rw)
dashboard:
Image: docker.io/gocrane/dashboard:v0.10.0
Port:
Host Port:
Environment:
Mounts:
/etc/nginx/conf.d/ from nginx-conf (rw)
Volumes:
cert:
Type: Secret (a volume populated by a Secret)
SecretName: webhook-server-tls
Optional: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: recommendation-configuration
Optional: false
nginx-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: nginx-conf
Optional: false
Conditions:
Type Status Reason

Available False MinimumReplicasUnavailable
Progressing True ReplicaSetUpdated
OldReplicaSets:
NewReplicaSet: craned-75d5fcff49 (1/1 replicas created)
Events:
Type Reason Age From Message

Normal ScalingReplicaSet 5m1s deployment-controller Scaled up replica set craned-75d5fcff49 to 1

@qmhu
Copy link
Member

qmhu commented May 7, 2023

Your k8s version is too high, you can use kindest/node:v1.21.1 to install a lower k8s cluster

@flying1574
Copy link
Author

I used the kindest/node:v1.21.1 , just I install the kubectl version is : v1.27.1
and I also try to use v1.25.0 version kubectl, also get the same result
you can check the details

[root@crane-master ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d22e37acb2f9 kindest/node:v1.21.1 "/usr/local/bin/entr…" 4 minutes ago Up 4 minutes 127.0.0.1:41058->6443/tcp crane-control-plane
[root@crane-master ~]# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-06-22T04:47:11Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.25) and server (1.21) exceeds the supported minor version skew of +/-1
[root@crane-master ~]# kubectl get deploy -n crane-system
NAME READY UP-TO-DATE AVAILABLE AGE
craned 0/1 1 0 4m46s
fadvisor 1/1 1 1 4m45s
grafana 1/1 1 1 4m47s
kube-state-metrics 1/1 1 1 4m48s
metric-adapter 1/1 1 1 4m46s
prometheus-server 1/1 1 1 4m48s
[root@crane-master ~]# kubectl get pods -n crane-system
NAME READY STATUS RESTARTS AGE
craned-75d5fcff49-k7z9q 1/2 CrashLoopBackOff 4 4m45s
fadvisor-6c6867dcb9-69n48 1/1 Running 0 4m45s
grafana-668f748bcd-s2qdk 1/1 Running 0 4m45s
kube-state-metrics-f4b6499d4-6h8g2 1/1 Running 0 4m45s
metric-adapter-789b5b8bc5-q2dhj 1/1 Running 0 4m45s
prometheus-server-6cb8bc86c4-vjq6f 2/2 Running 0 4m45s
[root@crane-master ~]# kubectl logs deploy/craned -n crane-system
Defaulted container "craned" out of: craned, dashboard
I0507 12:26:07.681381 1 nodelocal.go:25] Registering node local metrics collector cpu
I0507 12:26:07.681466 1 nodelocal.go:25] Registering node local metrics collector cpuLoad
I0507 12:26:07.681477 1 nodelocal.go:25] Registering node local metrics collector diskio
I0507 12:26:07.681486 1 nodelocal.go:25] Registering node local metrics collector memory
I0507 12:26:07.681495 1 nodelocal.go:25] Registering node local metrics collector netio
I0507 12:26:07.755677 1 predictor.go:121] predictors map[dsp:0xc000020c30 percentile:0xc0003afe00]
I0507 12:26:07.759844 1 webhook.go:103] Succeed to setup autoscaling webhook
I0507 12:26:07.760574 1 manager.go:142] Recommendation Config updated.
I0507 12:26:07.760956 1 manager.go:98] Start watching /tmp/recommendation-config/config.yaml for update.
I0507 12:26:07.764904 1 predictor.go:141] predictor manager started, all predictors started
I0507 12:26:07.765081 1 prediction.go:302] predictor Percentile started
I0507 12:26:07.765184 1 prediction.go:151] predictor Periodic started
I0507 12:26:07.765234 1 leaderelection.go:248] attempting to acquire leader lease crane-system/craned...
I0507 12:26:07.773635 1 server.go:94] install crane api server middleware: recovery
I0507 12:26:07.773661 1 server.go:94] install crane api server middleware: log
I0507 12:26:07.773669 1 server.go:94] install crane api server middleware: cors
I0507 12:26:07.774104 1 server.go:149] Start to listening on http address: 0.0.0.0:8082
I0507 12:26:07.776209 1 leaderelection.go:258] successfully acquired lease crane-system/craned
[root@crane-master ~]# kubectl describe deploy/craned -n crane-system
Name: craned
Namespace: crane-system
CreationTimestamp: Sun, 07 May 2023 12:24:41 +0800
Labels: app=craned
app.kubernetes.io/instance=crane
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=crane
app.kubernetes.io/version=v0.10.0
helm.sh/chart=crane-0.10.0
Annotations: deployment.kubernetes.io/revision: 1
meta.helm.sh/release-name: crane
meta.helm.sh/release-namespace: crane-system
Selector: app=craned
Replicas: 1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=craned
Service Account: craned
Containers:
craned:
Image: docker.io/gocrane/craned:v0.10.0
Port: 9443/TCP
Host Port: 0/TCP
Command:
/craned
Args:
--recommendation-configuration-file=/tmp/recommendation-config/config.yaml
--feature-gates=Analysis=true,TimeSeriesPrediction=true,Autoscaling=true,NodeResource=false
--prometheus-address=http://prometheus-server.crane-system.svc.cluster.local:8080
--v=2
Readiness: http-get http://:8082/api/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
TZ: Asia/Shanghai
CRANE_SYSTEM_NAMESPACE: (v1:metadata.namespace)
Mounts:
/tmp/k8s-webhook-server/serving-certs from cert (ro)
/tmp/recommendation-config from config (rw)
dashboard:
Image: docker.io/gocrane/dashboard:v0.10.0
Port:
Host Port:
Environment:
Mounts:
/etc/nginx/conf.d/ from nginx-conf (rw)
Volumes:
cert:
Type: Secret (a volume populated by a Secret)
SecretName: webhook-server-tls
Optional: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: recommendation-configuration
Optional: false
nginx-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: nginx-conf
Optional: false
Conditions:
Type Status Reason

Available False MinimumReplicasUnavailable
Progressing True ReplicaSetUpdated
OldReplicaSets:
NewReplicaSet: craned-75d5fcff49 (1/1 replicas created)
Events:
Type Reason Age From Message

Normal ScalingReplicaSet 4m57s deployment-controller Scaled up replica set craned-75d5fcff49 to 1
[root@crane-master ~]# kubectl logs pods/craned-75d5fcff49-k7z9q -n crane-system
Defaulted container "craned" out of: craned, dashboard
I0507 12:26:07.681381 1 nodelocal.go:25] Registering node local metrics collector cpu
I0507 12:26:07.681466 1 nodelocal.go:25] Registering node local metrics collector cpuLoad
I0507 12:26:07.681477 1 nodelocal.go:25] Registering node local metrics collector diskio
I0507 12:26:07.681486 1 nodelocal.go:25] Registering node local metrics collector memory
I0507 12:26:07.681495 1 nodelocal.go:25] Registering node local metrics collector netio
I0507 12:26:07.755677 1 predictor.go:121] predictors map[dsp:0xc000020c30 percentile:0xc0003afe00]
I0507 12:26:07.759844 1 webhook.go:103] Succeed to setup autoscaling webhook
I0507 12:26:07.760574 1 manager.go:142] Recommendation Config updated.
I0507 12:26:07.760956 1 manager.go:98] Start watching /tmp/recommendation-config/config.yaml for update.
I0507 12:26:07.764904 1 predictor.go:141] predictor manager started, all predictors started
I0507 12:26:07.765081 1 prediction.go:302] predictor Percentile started
I0507 12:26:07.765184 1 prediction.go:151] predictor Periodic started
I0507 12:26:07.765234 1 leaderelection.go:248] attempting to acquire leader lease crane-system/craned...
I0507 12:26:07.773635 1 server.go:94] install crane api server middleware: recovery
I0507 12:26:07.773661 1 server.go:94] install crane api server middleware: log
I0507 12:26:07.773669 1 server.go:94] install crane api server middleware: cors
I0507 12:26:07.774104 1 server.go:149] Start to listening on http address: 0.0.0.0:8082
I0507 12:26:07.776209 1 leaderelection.go:258] successfully acquired lease crane-system/craned
[root@crane-master ~]# kubectl describe pods/craned-75d5fcff49-k7z9q -n crane-system
Name: craned-75d5fcff49-k7z9q
Namespace: crane-system
Priority: 0
Service Account: craned
Node: crane-control-plane/172.18.0.2
Start Time: Sun, 07 May 2023 12:25:09 +0800
Labels: app=craned
pod-template-hash=75d5fcff49
Annotations:
Status: Running
IP: 10.244.0.10
IPs:
IP: 10.244.0.10
Controlled By: ReplicaSet/craned-75d5fcff49
Containers:
craned:
Container ID: containerd://4a22333d2418aa40a463d3f9be53331cab7c6fff574f06e920be80220cb772b6
Image: docker.io/gocrane/craned:v0.10.0
Image ID: docker.io/gocrane/craned@sha256:dccee54737586f43cd32c4f8b2588a8125396057e23b5c7d3cdcf71d3cdd515d
Port: 9443/TCP
Host Port: 0/TCP
Command:
/craned
Args:
--recommendation-configuration-file=/tmp/recommendation-config/config.yaml
--feature-gates=Analysis=true,TimeSeriesPrediction=true,Autoscaling=true,NodeResource=false
--prometheus-address=http://prometheus-server.crane-system.svc.cluster.local:8080
--v=2
State: Running
Started: Sun, 07 May 2023 12:26:07 +0800
Ready: True
Restart Count: 0
Readiness: http-get http://:8082/api/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
TZ: Asia/Shanghai
CRANE_SYSTEM_NAMESPACE: crane-system (v1:metadata.namespace)
Mounts:
/tmp/k8s-webhook-server/serving-certs from cert (ro)
/tmp/recommendation-config from config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cwldq (ro)
dashboard:
Container ID: containerd://1a2db773a3b35b9767dc972a021239926b00f1956abb3ca01f9385d7cd4de3ef
Image: docker.io/gocrane/dashboard:v0.10.0
Image ID: docker.io/gocrane/dashboard@sha256:7a17e6e50708df5449161baf3a9a5c99ce44ca887d2b9de64e86a2dcadcacc3f
Port:
Host Port:
State: Terminated
Reason: Error
Exit Code: 1
Started: Sun, 07 May 2023 12:30:01 +0800
Finished: Sun, 07 May 2023 12:30:01 +0800
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sun, 07 May 2023 12:28:29 +0800
Finished: Sun, 07 May 2023 12:28:29 +0800
Ready: False
Restart Count: 5
Environment:
Mounts:
/etc/nginx/conf.d/ from nginx-conf (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cwldq (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
cert:
Type: Secret (a volume populated by a Secret)
SecretName: webhook-server-tls
Optional: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: recommendation-configuration
Optional: false
nginx-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: nginx-conf
Optional: false
kube-api-access-cwldq:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

Warning FailedScheduling 5m7s (x3 over 5m23s) default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
Normal Scheduled 5m3s default-scheduler Successfully assigned crane-system/craned-75d5fcff49-k7z9q to crane-control-plane
Normal Pulling 4m59s kubelet Pulling image "docker.io/gocrane/craned:v0.10.0"
Normal Pulled 4m5s kubelet Successfully pulled image "docker.io/gocrane/craned:v0.10.0" in 54.489373268s
Normal Created 4m5s kubelet Created container craned
Normal Started 4m5s kubelet Started container craned
Normal Pulling 4m5s kubelet Pulling image "docker.io/gocrane/dashboard:v0.10.0"
Normal Pulled 3m5s kubelet Successfully pulled image "docker.io/gocrane/dashboard:v0.10.0" in 59.422633117s
Normal Created 2m25s (x4 over 3m5s) kubelet Created container dashboard
Normal Started 2m25s (x4 over 3m5s) kubelet Started container dashboard
Warning BackOff 118s (x7 over 3m4s) kubelet Back-off restarting failed container
Normal Pulled 103s (x4 over 3m5s) kubelet Container image "docker.io/gocrane/dashboard:v0.10.0" already present on machine

@flying1574
Copy link
Author

the problem is server dns,,you should set he DNS 8.8.8.8,and try installation
this is my problem and solution

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@qmhu @flying1574