Skip to content

perf: skip task queue metadata blob write and serialization in CreateTasks#9622

Open
mykaul wants to merge 2 commits intotemporalio:mainfrom
mykaul:perf/skip-taskqueue-blob-in-create-tasks
Open

perf: skip task queue metadata blob write and serialization in CreateTasks#9622
mykaul wants to merge 2 commits intotemporalio:mainfrom
mykaul:perf/skip-taskqueue-blob-in-create-tasks

Conversation

@mykaul
Copy link
Contributor

@mykaul mykaul commented Mar 23, 2026

Summary

  • Commit 1: Remove the TaskQueueInfo proto blob write from the Cassandra CreateTasks CAS batch. Only the range_id fencing token is needed for the conditional check — the full blob (~200-300 bytes) was written unnecessarily on every CreateTasks call.
  • Commit 2: Skip TaskQueueInfo blob serialization in the middle persistence layer, since no store backend needs it after commit 1.

Motivation

The CreateTasks path is the hottest write path in the matching service. Every task dispatch writes the full TaskQueueInfo proto blob as part of the CAS batch, but only range_id is used for fencing. This adds unnecessary serialization cost and increases Paxos/Raft proposal payload size.

This matches the SQL backend behavior, which already returns UpdatedMetadata: false and relies on SyncState to periodically flush metadata changes.

Approach

  • Introduces templateCheckRangeIDQuery that only sets range_id in the CAS UPDATE, removing task_queue and task_queue_encoding columns from the hot path.
  • The middle layer (task_manager.go) no longer serializes the TaskQueueInfo blob since no store consumes it.
  • Commit 2 depends on commit 1 — without the store-level change, the Cassandra V1 store would panic on nil blob access.

Testing

  • Cassandra persistence tests pass.
  • Matching service compiles cleanly.
  • Includes a new test TestCreateTask_DoesNotWriteTaskQueueMetadata verifying metadata is not written during CreateTasks.

mykaul added 2 commits March 23, 2026 11:29
The Cassandra CreateTasks CAS batch previously wrote the full TaskQueueInfo
proto blob (~200-300 bytes) on every call, but only the range_id fencing
token is needed for the conditional check. This matches the SQL backend
which already returns UpdatedMetadata:false and relies on SyncState to
periodically flush metadata changes.

Introduces templateCheckRangeIDQuery that only sets range_id in the CAS
UPDATE, removing the task_queue and task_queue_encoding columns from the
hot path. This reduces Paxos/Raft proposal payload size and eliminates
unnecessary TaskQueueInfo serialization writes.

Benchmark results (throughput_stress mc150, 5min, host networking,
GOMAXPROCS=4 GOGC=200, cores 0-3 server / 4-7 DB / 8-11 ES / 12-15 omes):
  Cassandra: before 192 / after 202 iterations (+5.2%)
  ScyllaDB:  before 171 / after 245 iterations (+43.3%)

Note: Temporal server CPUs (0-3) were >80% utilized in both runs, while DB
CPUs (4-7) were <65%, indicating the server is the throughput bottleneck.
@mykaul mykaul requested review from a team as code owners March 23, 2026 09:32
@mykaul
Copy link
Contributor Author

mykaul commented Mar 23, 2026

I've tested this extensively with ScyllaDB 2026.1, Cassandra 5.0 and Cassandra 3.11, running multiple Omes workloads - short periods of multiple workloads.

@yycptt yycptt requested a review from dnr March 23, 2026 18:56
@dnr
Copy link
Contributor

dnr commented Mar 25, 2026

Fyi, we're discussing this internally. There's no correctness concern, but there is some concern about metadata updates happening less frequently, mostly about the backlog size. We might want to do a compromise and skip this sometimes but not always.

Do you have any numbers to quantify the improvements you saw? How did you notice this issue in the first place?

@mykaul
Copy link
Contributor Author

mykaul commented Mar 25, 2026

Fyi, we're discussing this internally. There's no correctness concern, but there is some concern about metadata updates happening less frequently, mostly about the backlog size. We might want to do a compromise and skip this sometimes but not always.

Do you have any numbers to quantify the improvements you saw? How did you notice this issue in the first place?

AI noticed the potential improvement as I was working to improve the performance of ScyllaDB with Temporal (see #8652 ).
I'm testing every commit by itself, with all 3 DBs - see #9482 (comment) (I did not push all of them in order not to flood the team with PRs)

Do note that I do NOT feel I'm able to truly saturate the environment - it's either my pathetic laptop or the setup - or omes, which fails to properly load the system (ref #8652 (comment) )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants