Skip to content

Conversation

jakobht
Copy link
Member

@jakobht jakobht commented Oct 2, 2025

What changed?
We now have an in-memory cache for each of the name spaces on each shard distributor instance. The cache will be updated when things change in the database, using ETCD watches.

Also separated the ETCD store into more modules.

Why?
Before we attempted to push all the shard-to-executor mappings directly to the ETCD in a single transaction, this didn't work due to the number of transactions being way too big for ETCD.

How did you test it?
Unit tests and local runs

Potential risks
Currently the subscriptions to the database produces too many events, will address this in the next PR

Release notes

Documentation Changes

jakobht added 17 commits October 9, 2025 09:13
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
…th application logic

Signed-off-by: Jakob Haahr Taankvist <[email protected]>
… it's own package

Signed-off-by: Jakob Haahr Taankvist <[email protected]>
Signed-off-by: Jakob Haahr Taankvist <[email protected]>
@jakobht jakobht force-pushed the shardDistributorMemoryChacheRemoveShardToExecutorMapping branch from 22cfc2b to bc389d0 Compare October 9, 2025 07:14
@eleonoradgr eleonoradgr self-requested a review October 9, 2025 11:41
Copy link
Contributor

@eleonoradgr eleonoradgr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed together offline

Signed-off-by: Jakob Haahr Taankvist <[email protected]>
@jakobht jakobht merged commit 80a47da into cadence-workflow:master Oct 9, 2025
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants