Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Zookeeper backend. randomly not sync files after changes #876

Open
itshikanow opened this issue Jul 7, 2023 · 0 comments
Open

Zookeeper backend. randomly not sync files after changes #876

itshikanow opened this issue Jul 7, 2023 · 0 comments

Comments

@itshikanow
Copy link

Hello
I'm setup patroni cluster, and use confd to configure haproxy.
Backend zookeeper

Confd configs structure

/usr/local/etc/confd$ tree 
.
├── conf.d
│   └── haproxy.toml
├── confd.toml
└── templates
    └── haproxy.tmpl

cat confd.toml

# log-level = "info"
log-level = "debug"
backend = "zookeeper"
confdir = "/usr/local/etc/confd"
watch = true
interval = 1
nodes = ["192.168.50.1:2181",]
noop = false
prefix = "/patroni/15-main"

cat conf.d/haproxy.toml

[template]
# prefix = "/patroni/15-main/"
owner = "haproxy"
mode = "0644"
src = "haproxy.tmpl"
dest = "/usr/local/etc/haproxy/haproxy.cfg"
check_cmd = "/usr/local/sbin/haproxy -c -f {{ .src }}"
reload_cmd = "/bin/bash -c 'true'"
keys = ["/members",]

cat templates/haproxy.tmpl

global
        maxconn 1000
        log stdout format raw local0
        # user haproxy
        # group haproxy

defaults
        log     global
        mode    tcp
        retries 2
        timeout client 120m
        timeout server 120m
        timeout connect 4s
        timeout check 5s
#       balance leastconn
        option tcplog
        option log-health-checks

listen stats
        mode http
        bind *:7000
        stats enable
        stats uri /

listen master
        bind *:5428
        option httpchk OPTIONS /master
        http-check expect status 200
        default-server inter 2s fall 3 rise 2 on-marked-down shutdown-sessions
{{- range gets "/members/*" }}
{{- $data := json .Value }}
        server {{base .Key}} {{base (replace (index (split $data.conn_url "/") 2) "@" "/" -1)}} maxconn 300 check port {{index (split (index (split $data.api_url "/") 2) ":") 1}}
{{- end }}
listen replicas
        bind *:5429
        option httpchk OPTIONS /replica?lag=10MB
        http-check expect status 200
        default-server inter 2s fall 3 rise 2 on-marked-down shutdown-sessions
{{- range gets "/members/*" }}
{{- $data := json .Value }}
        server {{base .Key}} {{base (replace (index (split $data.conn_url "/") 2) "@" "/" -1)}} maxconn 300 check port {{index (split (index (split $data.api_url "/") 2) ":") 1}}
{{- end }}
{{- range gets "/members/*" }}                                             
{{- $data := json .Value }}
{{- if eq "master" $data.role }}
        server {{base .Key}}-master {{base (replace (index (split $data.conn_url "/") 2) "@" "/" -1)}} maxconn 300 backup
{{- end }}
{{- end }}

start command:

confd -config-file  /usr/local/etc/confd/confd.toml

I'm start confd, waiting few seconds. Cond generate haproxy config right. I'm stoping one node of patroni, waiting 1 minute, to disapear znode from zookeeper and config has changed exptectialy, the one node disapear from haproxy config

After this i'm start the patroni node, at first config changed right. But after few seconds config reverted to state without second patroni node while in zookeeper all 2 znode exist and have right data.

If restart confd at this point, the config will regenerate right.

If change interval to from 1 to 5 this behaviour not so frequent, but repeats. I'm need confidence that confd work right.

If use -oneshot and start confd in bash script all work well. But spawn avery 1 second new process is bad.

here the last state of zookeeper after enable patroni node back:
ls /patroni/15-main/members

[devlocal20, devlocal20-2]

get /patroni/15-main/members/devlocal20

{"conn_url":"postgres://192.168.50.200:5440/postgres","api_url":"http://192.168.50.200:8008/patroni","state":"running","role":"replica","version":"3.0.1","xlog_location":4429186584,"timeline":11}

get /patroni/15-main/members/devlocal20-2

{"conn_url":"postgres://192.168.50.201:5440/postgres","api_url":"http://192.168.50.201:8008/patroni","state":"running","role":"master","version":"3.0.1","xlog_location":4429186584,"timeline":11}

Here the full log of confd with this behaviour

2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: INFO Backend set to zookeeper
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: INFO Starting confd
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: INFO Backend source(s) set to 192.168.50.1:2181
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Loading template resources from confdir /usr/local/etc/confd
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Found template: /usr/local/etc/confd/conf.d/haproxy.toml
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Loading template resource from /usr/local/etc/confd/conf.d/haproxy.toml
2023/07/07 13:55:58 Connected to 192.168.50.1:2181
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Retrieving keys from store
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Key prefix set to /patroni/15-main
2023/07/07 13:55:58 Authenticated: id=72058294651454940, timeout=4000
2023/07/07 13:55:58 Re-submitting `0` credentials after reconnect
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Got the following map from store: map[/patroni/15-main/members/devlocal20:{"conn_url":"postgres://192.168.50.200:5440/postgres","api_url":"http://192.168.50.200:8008/patroni","state":"running","role":"replica","version":"3.0.1","xlog_location":4429186584,"timeline":11} /patroni/15-main/members/devlocal20-2:{"conn_url":"postgres://192.168.50.201:5440/postgres","api_url":"http://192.168.50.201:8008/patroni","state":"running","role":"master","version":"3.0.1","xlog_location":4429186584,"timeline":11}]
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Using source template /usr/local/etc/confd/templates/haproxy.tmpl
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Compiling source template /usr/local/etc/confd/templates/haproxy.tmpl
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Comparing candidate config to /usr/local/etc/haproxy/haproxy.cfg
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: INFO /usr/local/etc/haproxy/haproxy.cfg has md5sum d41d8cd98f00b204e9800998ecf8427e should be 910dff18d217291b885650402dd3efeb
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: INFO Target config /usr/local/etc/haproxy/haproxy.cfg out of sync
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Running /usr/local/sbin/haproxy -c -f /usr/local/etc/haproxy/.haproxy.cfg670469573
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG "Configuration file is valid\n"
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Overwriting target config /usr/local/etc/haproxy/haproxy.cfg
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Running /bin/bash -c 'true'
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG ""
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: INFO Target config /usr/local/etc/haproxy/haproxy.cfg has been updated
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members/devlocal20-2
2023-07-07T13:55:58+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members/devlocal20

2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Retrieving keys from store
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main/members
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Key prefix set to /patroni/15-main
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main/members/devlocal20-2
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main/members/devlocal20
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Got the following map from store: map[/patroni/15-main/members/devlocal20-2:{"conn_url":"postgres://192.168.50.201:5440/postgres","api_url":"http://192.168.50.201:8008/patroni","state":"running","role":"master","version":"3.0.1","xlog_location":4429186584,"timeline":11}]
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Using source template /usr/local/etc/confd/templates/haproxy.tmpl
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Compiling source template /usr/local/etc/confd/templates/haproxy.tmpl
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Comparing candidate config to /usr/local/etc/haproxy/haproxy.cfg
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: INFO /usr/local/etc/haproxy/haproxy.cfg has md5sum 910dff18d217291b885650402dd3efeb should be 1ef749761af34e16ebe67cff2ae4d849
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: INFO Target config /usr/local/etc/haproxy/haproxy.cfg out of sync
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Running /usr/local/sbin/haproxy -c -f /usr/local/etc/haproxy/.haproxy.cfg861808736
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG "Configuration file is valid\n"
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Overwriting target config /usr/local/etc/haproxy/haproxy.cfg
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Running /bin/bash -c 'true'
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG ""
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: INFO Target config /usr/local/etc/haproxy/haproxy.cfg has been updated
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni
2023-07-07T13:56:45+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members/devlocal20-2

2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Retrieving keys from store
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main/members
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Key prefix set to /patroni/15-main
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main/members/devlocal20-2
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Got the following map from store: map[/patroni/15-main/members/devlocal20:{"conn_url":"postgres://192.168.50.200:5440/postgres","api_url":"http://192.168.50.200:8008/patroni","state":"stopped","role":"replica","version":"3.0.1"} /patroni/15-main/members/devlocal20-2:{"conn_url":"postgres://192.168.50.201:5440/postgres","api_url":"http://192.168.50.201:8008/patroni","state":"running","role":"master","version":"3.0.1","xlog_location":4429186584,"timeline":11}]
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Using source template /usr/local/etc/confd/templates/haproxy.tmpl
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Compiling source template /usr/local/etc/confd/templates/haproxy.tmpl
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Comparing candidate config to /usr/local/etc/haproxy/haproxy.cfg
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: INFO /usr/local/etc/haproxy/haproxy.cfg has md5sum 1ef749761af34e16ebe67cff2ae4d849 should be 910dff18d217291b885650402dd3efeb
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: INFO Target config /usr/local/etc/haproxy/haproxy.cfg out of sync
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Running /usr/local/sbin/haproxy -c -f /usr/local/etc/haproxy/.haproxy.cfg303199807
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG "Configuration file is valid\n"
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Overwriting target config /usr/local/etc/haproxy/haproxy.cfg
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Running /bin/bash -c 'true'
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG ""
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: INFO Target config /usr/local/etc/haproxy/haproxy.cfg has been updated
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members/devlocal20
2023-07-07T13:56:53+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members/devlocal20-2
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Retrieving keys from store
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main/members/devlocal20
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main/members/devlocal20-2
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Key prefix set to /patroni/15-main
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Stop watching: /patroni/15-main/members
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Got the following map from store: map[/patroni/15-main/members/devlocal20-2:{"conn_url":"postgres://192.168.50.201:5440/postgres","api_url":"http://192.168.50.201:8008/patroni","state":"running","role":"master","version":"3.0.1","xlog_location":4429186584,"timeline":11}]
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Using source template /usr/local/etc/confd/templates/haproxy.tmpl
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Compiling source template /usr/local/etc/confd/templates/haproxy.tmpl
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Comparing candidate config to /usr/local/etc/haproxy/haproxy.cfg
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: INFO /usr/local/etc/haproxy/haproxy.cfg has md5sum 910dff18d217291b885650402dd3efeb should be 1ef749761af34e16ebe67cff2ae4d849
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: INFO Target config /usr/local/etc/haproxy/haproxy.cfg out of sync
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Running /usr/local/sbin/haproxy -c -f /usr/local/etc/haproxy/.haproxy.cfg080067730
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG "Configuration file is valid\n"
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Overwriting target config /usr/local/etc/haproxy/haproxy.cfg
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Running /bin/bash -c 'true'
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG ""
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: INFO Target config /usr/local/etc/haproxy/haproxy.cfg has been updated
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members/devlocal20
2023-07-07T13:56:57+03:00 ubuntu16-work confd[196]: DEBUG Watching: /patroni/15-main/members/devlocal20-2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@itshikanow