"reconfig" always returns BadVersionError when "from_config" is set, even if it is set correctly

[zbentley]

Summary: Kazoo appears to be supplying "from_config"/"config_version" incorrectly in the reconfig command.

Expected Behavior

1. get('/zookeeper/config') yields a config with a version.
2. reconfig(joining"some hosts", leaving=None, new_hosts=None, from_config=version), where version is the version from step 1, works correctly.
3. If reconfig fails with BadVersionError, any subsequent get('/zookeeper/config') calls should return a different version.
4. However, if I do echo 'reconfig -v THE_VERSION -add SOME_HOSTS | zkCli.sh, the reconfiguration succeeds, with the same version string as above.

Actual Behavior

1. get('/zookeeper/config') yields a config with a version.
2. reconfig(joining"some hosts", leaving=None, new_hosts=None, from_config=version), always returns BadVersionError whenever from_config is supplied. Only omitting from_config allows reconfiguration to succeed.
3. After getting a BadVersionError, subsequent get('/zookeeper/config') return the same version as returned in step 1.

Additionally, the int type assertion for from_config seems incorrect, as config is a zxid type (documented here), meaning that values like 1000000d5 are valid.

Suspected Root Cause

I think that Kazoo is incorrectly serializing the from_config value as an int instead of a zxid, though I'm not quite sure how to serialize it correctly.

An erroneous type assertion is here, and erroneous serialization code is here.

However, removing the type assertion and changing the serialization code to write_string did not resolve the problem for me; I was able to supply a str to client.reconfig without failing the type assertion, but still received a BadVersionError.

Snippet to Reproduce the Problem

def parse_zk_config(raw):
    for line in raw.splitlines():
        line = line.strip()
        if line.startswith('version'):
            return line.split('=', 1)[-1]

c = KazooClient()
c.start()
version = parse_zk_config(c.get('/zookeeper/config')[0])
version = int(version)  # Fails unless ZK version happens to be an int-castable value
c.reconfig(joining='server.3=22.0.46.63:2182:2183:participant;2181', leaving=None, new_members=None, config_id=version)

That snippet always fails for me (BadVersionError on last line). No other reconfigurations occur with my test ensemble, so there's not a race (i.e. the BadVersionError). Subsequent calls to c.get('/zookeeper/config') confirm that the version isn't changing.

Furthermore, the following command succeeds (assuming an initial version value of 100000000):

echo 'reconfig -v 100000000 -add server.3=22.0.46.63:2182:2183:participant;2181' | zkCli.sh

Logs with logging in DEBUG mode

DEBUG:kazoo.client:Sending request(xid=1): GetData(path='/zookeeper/config', watcher=None)
DEBUG:kazoo.client:Received response(xid=1): ('server.1=22.0.29.211:2182:2183:participant;0.0.0.0:2181\nversion=100000000', ZnodeStat(czxid=0, mzxid=4294967296, ctime=0, mtime=1604673499180, version=-1, cversion=0, aversion=-1, ephemeralOwner=0, dataLength=73, numChildren=0, pzxid=0))
DEBUG:kazoo.client:Sending request(xid=2): Reconfig(joining='server.3=22.0.46.63:2182:2183:participant;2181', leaving=None, new_members=None, config_id='100000000')
DEBUG:kazoo.client:Received error(xid=2) BadVersionError((), {})

Specifications

• Kazoo version: 2.8.0
• Result of pip list command:

ami-build-and-cloud-init-scripts 0.0.1  ## proprietary
awscli                           1.16.314
boto3                            1.10.50
botocore                         1.13.50
certifi                          2020.6.20
chardet                          3.0.4
colorama                         0.4.1
docutils                         0.16
futures                          3.3.0
idna                             2.9
Jinja2                           2.11.2
jmespath                         0.10.0
kazoo                            2.2.1
MarkupSafe                       1.1.1
monotonic                        1.5
pip                              20.1
pyasn1                           0.4.8
python-dateutil                  2.8.1
PyYAML                           5.2
requests                         2.23.0
rsa                              3.4.2
s3transfer                       0.2.1
sentry-sdk                       0.7.10
setuptools                       18.5
six                              1.15.0
tenacity                         6.2.0
typing                           3.7.4.3
urllib3                          1.25.9
wheel                            0.34.2

• Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
• Zookeeper configuration: put here any useful ZK configuration: Stock configuration, with the following customizations:

4lw.commands.whitelist=*
skipACL=yes  # To allow reconfig
standaloneEnabled=false
dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic

• Python version: 2.7.17
• OS: Ubuntu 18.04.4 LTS