-
Notifications
You must be signed in to change notification settings - Fork 20
Fetching lease fails with TTL #1
Comments
@hemna in the
|
I'm running a single devstack deployment on 2 separate nodes to try and test out Cinder's active/active HA capability with the 3PAR cinder driver. I run into this while running some rally scenarios. stack@walt-stack-2 /opt/stack/bin ./etcd --version |
@hemna As its going to be hard to recreate, could you please log the response we get from |
Seems to happen in gate sporadically, see http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22KeyError%3A%20'TTL'%5C%22 From the timestamps it might be related to etcd getting slow: |
Looks like they have a FAQ just for this scenario : https://coreos.com/etcd/docs/latest/faq.html#performance |
I'm using etc3dgw in openstack cinder and I'm running into an issue where the lease refresh is failing, due to a missing field.
stack@walt-stack-1 /opt/stack/logs/screen pip freeze |grep etcd 1 ↵ 482 12:23:14
etcd3==0.6.2
etcd3gw==0.1.0
---excerpt from the cinder log
2017-08-23 12:25:16.771 24793 ERROR root [-] Unexpected exception occurred 60 time(s)... retrying.: KeyError: 'TTL'
2017-08-23 12:25:16.771 24793 ERROR root Traceback (most recent call last):
2017-08-23 12:25:16.771 24793 ERROR root File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 250, in wrapper
2017-08-23 12:25:16.771 24793 ERROR root return infunc(*args, **kwargs)
2017-08-23 12:25:16.771 24793 ERROR root File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 187, in _beat_forever_until_stopped
2017-08-23 12:25:16.771 24793 ERROR root wait_until_next_beat = self._driver.heartbeat()
2017-08-23 12:25:16.771 24793 ERROR root File "/usr/lib/python2.7/site-packages/tooz/drivers/etcd3gw.py", line 191, in heartbeat
2017-08-23 12:25:16.771 24793 ERROR root lock.heartbeat()
2017-08-23 12:25:16.771 24793 ERROR root File "/usr/lib/python2.7/site-packages/tooz/drivers/etcd3gw.py", line 38, in wrapper
2017-08-23 12:25:16.771 24793 ERROR root return func(*args, **kwargs)
2017-08-23 12:25:16.771 24793 ERROR root File "/usr/lib/python2.7/site-packages/tooz/drivers/etcd3gw.py", line 154, in heartbeat
2017-08-23 12:25:16.771 24793 ERROR root self._lease.refresh()
2017-08-23 12:25:16.771 24793 ERROR root File "/usr/lib/python2.7/site-packages/etcd3gw/lease.py", line 64, in refresh
2017-08-23 12:25:16.771 24793 ERROR root return int(result['result']['TTL'])
2017-08-23 12:25:16.771 24793 ERROR root KeyError: 'TTL'
2017-08-23 12:25:16.771 24793 ERROR root
The text was updated successfully, but these errors were encountered: