Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wire-server installation failed #470

Open
JonnyX11 opened this issue Jun 13, 2021 · 10 comments
Open

wire-server installation failed #470

JonnyX11 opened this issue Jun 13, 2021 · 10 comments

Comments

@JonnyX11
Copy link

Hello,

I got this error, Anyone know what to do ?

bash-4.4# helm upgrade --install wire-server wire/wire-server -f my-wire-server/values.yaml -f my-wire-server/secrets.yaml --wait

Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition

Can you please help?

@arthurwolf
Copy link
Contributor

arthurwolf commented Jun 13, 2021 via email

@JonnyX11
Copy link
Author

bash-4.4# helm upgrade --install wire-server wire/wire-server -f my-wire-server/values.yaml -f my-wire-server/secrets.yaml --wait --debug
history.go:56: [debug] getting history for release wire-server
upgrade.go:123: [debug] preparing upgrade for wire-server
upgrade.go:131: [debug] performing update for wire-server
upgrade.go:303: [debug] creating upgraded release for wire-server
client.go:282: [debug] Starting delete for "cassandra-migrations" Job
client.go:122: [debug] creating 1 resource(s)
client.go:491: [debug] Watching for changes to Job cassandra-migrations with timeout of 5m0s
client.go:519: [debug] Add/Modify event for cassandra-migrations: ADDED
client.go:558: [debug] cassandra-migrations: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:519: [debug] Add/Modify event for cassandra-migrations: MODIFIED
client.go:558: [debug] cassandra-migrations: Jobs active: 1, jobs failed: 0, jobs succeeded: 0

upgrade.go:369: [debug] warning: Upgrade "wire-server" failed: pre-upgrade hooks failed: timed out waiting for the condition
Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition
helm.go:81: [debug] pre-upgrade hooks failed: timed out waiting for the condition
UPGRADE FAILED
main.newUpgradeCmd.func2
helm.sh/helm/v3/cmd/helm/upgrade.go:157
github.com/spf13/cobra.(*Command).execute
github.com/spf13/[email protected]/command.go:850
github.com/spf13/cobra.(*Command).ExecuteC
github.com/spf13/[email protected]/command.go:958
github.com/spf13/cobra.(*Command).Execute
github.com/spf13/[email protected]/command.go:895
main.main
helm.sh/helm/v3/cmd/helm/helm.go:80
runtime.main
runtime/proc.go:225
runtime.goexit
runtime/asm_amd64.s:1371

@JonnyX11
Copy link
Author

JonnyX11 commented Jun 15, 2021

bash-4.4# helm upgrade --install wire-server wire/wire-server -f my-wire-server/values.yaml -f my-wire-server/secrets.yaml --wait --debug
history.go:56: [debug] getting history for release wire-server
upgrade.go:123: [debug] preparing upgrade for wire-server
upgrade.go:131: [debug] performing update for wire-server
upgrade.go:303: [debug] creating upgraded release for wire-server
client.go:282: [debug] Starting delete for "cassandra-migrations" Job
client.go:122: [debug] creating 1 resource(s)
client.go:491: [debug] Watching for changes to Job cassandra-migrations with timeout of 5m0s
client.go:519: [debug] Add/Modify event for cassandra-migrations: ADDED
client.go:558: [debug] cassandra-migrations: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:519: [debug] Add/Modify event for cassandra-migrations: MODIFIED
client.go:558: [debug] cassandra-migrations: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:519: [debug] Add/Modify event for cassandra-migrations: MODIFIED
client.go:282: [debug] Starting delete for "elasticsearch-index-create" Job
client.go:122: [debug] creating 1 resource(s)
client.go:491: [debug] Watching for changes to Job elasticsearch-index-create with timeout of 5m0s
client.go:519: [debug] Add/Modify event for elasticsearch-index-create: ADDED
client.go:558: [debug] elasticsearch-index-create: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:519: [debug] Add/Modify event for elasticsearch-index-create: MODIFIED
client.go:558: [debug] elasticsearch-index-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
upgrade.go:369: [debug] warning: Upgrade "wire-server" failed: pre-upgrade hooks failed: timed out waiting for the condition
Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition
helm.go:81: [debug] pre-upgrade hooks failed: timed out waiting for the condition
UPGRADE FAILED
main.newUpgradeCmd.func2
helm.sh/helm/v3/cmd/helm/upgrade.go:157
github.com/spf13/cobra.(*Command).execute
github.com/spf13/[email protected]/command.go:850
github.com/spf13/cobra.(*Command).ExecuteC
github.com/spf13/[email protected]/command.go:958
github.com/spf13/cobra.(*Command).Execute
github.com/spf13/[email protected]/command.go:895
main.main
helm.sh/helm/v3/cmd/helm/helm.go:80
runtime.main
runtime/proc.go:225
runtime.goexit
runtime/asm_amd64.s:1371

@JonnyX11
Copy link
Author

Anyone :(

@arthurwolf
Copy link
Contributor

Are you installing the demo stuff, or the production stuff? Which pages in the documentation are you following exactly? Can you give us a complete log of the commands you type, and their output, from the beginning?

I don't have an idea right now of what might be going wrong, but providing this information might help somebody else find a solution for you easier than without that information.

Sorry for taking so long to answer, I completely missed the notification for your message in my inbox.

@arthurwolf
Copy link
Contributor

How much RAM and how many CPUs do you have on the server? Can you give us the results of:

cat /proc/cpuinfo | grep processor | wc -l

free -m

As well as the result of «uptime» before and after the helm upgrade command ?

Thanks.

@lucendio
Copy link
Contributor

Hello @JonnyX11

we are always happy to help, but we kindly ask people who reach out for help to be cooperative. So, would you mind following the template structure that we put in place for these kind of scenarios? You might notice some significant overlap between the questions lisyted in the template and the ones asked by @arthurwolf

Additionally to what is in the template, please provide us with some more context. For example, is it your first installation or are you trying to upgrade an installation? If the latter, did you check the release notes for the new version you installed - also for wire-server? Please note, you will find the code for each chart in there.

Lastly, please provide us with the output of kubectl get pods -o wide --all-namespaces.

Thank you.

@JonnyX11
Copy link
Author

JonnyX11 commented Jun 20, 2021

first of all thank you very much for your help!

Basic information

  • On-premises: Yes

  • Cloud-Provider: nope

  • Installation type: production

  • Kubernetes version:
    bash-4.4# kubectl version
    Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

  • Helm version:
    bash-4.4# helm version
    version.BuildInfo{Version:"v3.5.2", GitCommit:"", GitTreeState:"", GoVersion:"go1.16"}

  • Installed with Kubespray:

  • (Helm) Charts version:
    bash-4.4# helm show chart wire/wire-server | grep -m 1 version
    version: 2.108.0

  • List of installed top-level charts:

  • Other related technologies + version:

bash-4.4# ansible --version
ansible 2.9.12
config file = /wire-server-deploy/ansible/ansible.cfg
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /nix/store/407c5v2x920kri79w0sb9gb1g82q28mg-python3.8-ansible-2.9.12/lib/python3.8/site-packages/ansible
executable location = /nix/store/407c5v2x920kri79w0sb9gb1g82q28mg-python3.8-ansible-2.9.12/bin/ansible
python version = 3.8.8 (default, Feb 19 2021, 11:04:50) [GCC 10.2.0]

How did you install Wire?

according to the productions docs:
https://docs.wire.com/how-to/install/version-requirements.html#persistence
https://docs.wire.com/how-to/install/helm-prod.html#helm-prod

I used the docker option for helm
https://docs.wire.com/how-to/install/dependencies.html#dependencies

I worked on Ubuntu 18.04 for all virtual machines

Name Amount CPU Cores Memory (GB) Disk Space (GB)
Cassandra 3 2 4 80
MinIO 3 1 2 400
ElasticSearch 3 1 2 60
Kubernetes³ 3 8 40
Restund⁴ 2 1 2 10
Per-Server Totals 11 CPU Cores 18 GB Memory 590 GB Disk Space
Admin Host² 1 1 4 40
Asset Host² 1 1 4 100
Per-Server Totals with Admin and Asset Hosts 13 CPU Cores 26 GB Memory 730 GB Disk Space

Please see the outputs here :

bash-4.4# kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

bash-4.4# kubectl get pods -o wide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default cassandra-migrations-d5g22 0/1 Completed 0 5m19s 10.233.65.23 kubenode02
default demo-smtp-7f7cc8b585-bcrkq 1/1 Running 4 10d 10.233.64.71 kubenode01
default elasticsearch-index-create-vm6zs 0/1 Init:0/1 3 5m14s 10.233.65.24 kubenode02
default fake-aws-s3-77d9447b8f-2vzgj 1/1 Running 0 66m 10.233.65.19 kubenode02
default fake-aws-s3-reaper-78d9f58dd4-zmqsb 1/1 Running 0 66m 10.233.66.2 kubenode03
default fake-aws-sns-6c7c4b7479-c4thd 2/2 Running 8 10d 10.233.64.74 kubenode01
default fake-aws-sqs-59fbfbcbd4-f7d72 2/2 Running 8 10d 10.233.64.72 kubenode01
default redis-ephemeral-master-0 1/1 Running 4 10d 10.233.64.73 kubenode01
kube-system coredns-7677f9bb54-4kg2d 1/1 Running 16 10d 10.233.65.18 kubenode02
kube-system coredns-7677f9bb54-cnxm2 1/1 Running 4 10d 10.233.64.70 kubenode01
kube-system dns-autoscaler-5b7b5c9b6f-vxwcw 1/1 Running 4 10d 10.233.64.75 kubenode01
kube-system kube-apiserver-kubenode01 1/1 Running 4 10d 172.16.25.80 kubenode01
kube-system kube-apiserver-kubenode02 1/1 Running 18 7d5h 172.16.25.83 kubenode02
kube-system kube-apiserver-kubenode03 1/1 Running 15 7d5h 172.16.25.97 kubenode03
kube-system kube-controller-manager-kubenode01 1/1 Running 0 10d 172.16.25.80 kubenode01
kube-system kube-controller-manager-kubenode02 1/1 Running 0 7d5h 172.16.25.83 kubenode02
kube-system kube-controller-manager-kubenode03 1/1 Running 0 7d5h 172.16.25.97 kubenode03
kube-system kube-flannel-dwpxs 1/1 Running 4 10d 172.16.25.80 kubenode01
kube-system kube-flannel-jdtgg 1/1 Running 16 7d5h 172.16.25.83 kubenode02
kube-system kube-flannel-lv8kw 1/1 Running 13 7d5h 172.16.25.97 kubenode03
kube-system kube-proxy-4k4qz 1/1 Running 0 3h34m 172.16.25.80 kubenode01
kube-system kube-proxy-4njtd 1/1 Running 0 3h34m 172.16.25.83 kubenode02
kube-system kube-proxy-dlhcc 1/1 Running 0 3h34m 172.16.25.97 kubenode03
kube-system kube-scheduler-kubenode01 1/1 Running 0 10d 172.16.25.80 kubenode01
kube-system kube-scheduler-kubenode02 1/1 Running 0 7d5h 172.16.25.83 kubenode02
kube-system kube-scheduler-kubenode03 1/1 Running 0 7d5h 172.16.25.97 kubenode03
kube-system nodelocaldns-gfvb7 1/1 Running 15 7d5h 172.16.25.97 kubenode03
kube-system nodelocaldns-jr6ft 1/1 Running 16 7d5h 172.16.25.83 kubenode02
kube-system nodelocaldns-vwhzg 1/1 Running 4 10d 172.16.25.80 kubenode01

bash-4.4# cat /proc/cpuinfo | grep processor | wc -l
4
bash-4.4#
bash-4.4# free -m
total used free shared buff/cache available
Mem: 16013 579 12578 1 2855 15135
Swap: 2047 0 2047
bash-4.4#

bash-4.4# helm upgrade --install wire-server wire/wire-server -f my-wire-server/values.yaml -f my-wire-server/secrets.yaml --wait --debug
history.go:56: [debug] getting history for release wire-server
upgrade.go:123: [debug] preparing upgrade for wire-server
upgrade.go:131: [debug] performing update for wire-server
upgrade.go:303: [debug] creating upgraded release for wire-server
client.go:282: [debug] Starting delete for "cassandra-migrations" Job
client.go:122: [debug] creating 1 resource(s)
client.go:491: [debug] Watching for changes to Job cassandra-migrations with timeout of 5m0s
client.go:519: [debug] Add/Modify event for cassandra-migrations: ADDED
client.go:558: [debug] cassandra-migrations: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:519: [debug] Add/Modify event for cassandra-migrations: MODIFIED
client.go:558: [debug] cassandra-migrations: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:519: [debug] Add/Modify event for cassandra-migrations: MODIFIED
client.go:282: [debug] Starting delete for "elasticsearch-index-create" Job
client.go:122: [debug] creating 1 resource(s)
client.go:491: [debug] Watching for changes to Job elasticsearch-index-create with timeout of 5m0s
client.go:519: [debug] Add/Modify event for elasticsearch-index-create: ADDED
client.go:558: [debug] elasticsearch-index-create: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:519: [debug] Add/Modify event for elasticsearch-index-create: MODIFIED
client.go:558: [debug] elasticsearch-index-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
upgrade.go:369: [debug] warning: Upgrade "wire-server" failed: pre-upgrade hooks failed: timed out waiting for the condition
Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition
helm.go:81: [debug] pre-upgrade hooks failed: timed out waiting for the condition
UPGRADE FAILED
main.newUpgradeCmd.func2
helm.sh/helm/v3/cmd/helm/upgrade.go:157
github.com/spf13/cobra.(*Command).execute
github.com/spf13/[email protected]/command.go:850
github.com/spf13/cobra.(*Command).ExecuteC
github.com/spf13/[email protected]/command.go:958
github.com/spf13/cobra.(*Command).Execute
github.com/spf13/[email protected]/command.go:895
main.main
helm.sh/helm/v3/cmd/helm/helm.go:80
runtime.main
runtime/proc.go:225
runtime.goexit
runtime/asm_amd64.s:1371

bash-4.4# helm list --all
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
cassandra-external default 2 2021-06-20 11:43:21.448671888 +0000 UTC deployed cassandra-external-2.108.0
databases-ephemeral default 2 2021-06-20 11:43:41.528915369 +0000 UTC deployed databases-ephemeral-2.108.0
elasticsearch-external default 2 2021-06-20 11:43:33.670740997 +0000 UTC deployed elasticsearch-external-2.108.0
fake-aws default 2 2021-06-20 11:49:41.841649782 +0000 UTC deployed fake-aws-2.108.0
minio-external default 4 2021-06-20 11:45:08.514165984 +0000 UTC deployed minio-external-2.108.0
smtp default 2 2021-06-20 11:54:14.683598062 +0000 UTC deployed demo-smtp-2.108.0 1.0
wire-server default 9 2021-06-20 12:50:43.972757468 +0000 UTC failed wire-server-2.108.0

@JonnyX11
Copy link
Author

JonnyX11 commented Jun 23, 2021

Hello @JonnyX11

we are always happy to help, but we kindly ask people who reach out for help to be cooperative. So, would you mind following the template structure that we put in place for these kind of scenarios? You might notice some significant overlap between the questions lisyted in the template and the ones asked by @arthurwolf

Additionally to what is in the template, please provide us with some more context. For example, is it your first installation or are you trying to upgrade an installation? If the latter, did you check the release notes for the new version you installed - also for wire-server? Please note, you will find the code for each chart in there.

Lastly, please provide us with the output of kubectl get pods -o wide --all-namespaces.

Thank you.

The details was OK on the last post ? it was clear ?

@JonnyX11
Copy link
Author

JonnyX11 commented Jul 1, 2021

Anyone?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants