Skip to content

feat(orchestrator): carry the queue explicitly on every internal payload - #527

Merged
behinddwalls merged 1 commit into
mainfrom
preetam/queue-on-payloads
Aug 6, 2026
Merged

feat(orchestrator): carry the queue explicitly on every internal payload#527
behinddwalls merged 1 commit into
mainfrom
preetam/queue-on-payloads

Conversation

@behinddwalls

@behinddwalls behinddwalls commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

Why?

Consumers today learn a message's queue only by loading the referenced entity from storage, and entity IDs are opaque — nothing may parse a queue out of an ID prefix. For storage to become queue-scoped (resolved per queue like every other extension), every consumer must hold the queue before its first storage read, so the queue has to ride on the wire explicitly.

What?

The internal payload types (RequestID, BatchID, BuildID, CancelRequest) gain a queue field; the change is additive JSON, and payloads written before the field existed decode with an empty queue. Publishers stamp it from the entity they already hold: start, mergeconflictsignal, batch, speculate, mergesignal, build, buildsignal, and the orchestrator cancel controller; the gateway cancel controller stamps the authoritative queue from the stored request summary, overriding caller input. Consumers guard that a non-empty payload queue matches the loaded entity's queue and reject mismatches as malformed (non-retryable, straight to DLQ) — the guard becomes the routing input once storage is queue-resolved. Buildsignal's re-publish to speculate now partitions by the batch's queue instead of the inherited batch-ID partition key, matching every other speculate publisher and restoring the per-queue serial-processing guarantee that stage relies on.

Test Plan

go test ./...make fmtmake lint — new unit tests cover the queue stamp on published payloads and the mismatch rejection guard.

behinddwalls added a commit that referenced this pull request Aug 6, 2026
…and writes (#525)

## Summary

### Why?

`BatchStore.GetByQueueAndStates` was the storage contract's last
query-by-attribute, backed by the schema's only secondary index
(`idx_queue_state`). The queue's per-state membership records
(`QueueBatchStateStore`) exist to replace it with pure by-key reads, but
had no callers yet.

### What?

Every batch state write now goes through the shared transition protocol
in `submitqueue/core/batch`: `Transition` in the batch
(Creating→Created), speculate (Speculating/Merging/Failed/Cancelled),
mergesignal (terminal), cancel (Cancelling), and DLQ (Failed)
controllers, with `EnsureRecord` repairing records after batch creation
and on idempotent redelivery skip branches. The batch controller's
dependency read goes through `core/batch.ListByStates` over the record
buckets. `GetByQueueAndStates` is deleted from the contract, MySQL
implementation, and mock, and `idx_queue_state` is dropped from the
batch schema, so the batch table is pure get/put-by-key. Also repairs
three stale controller tests that no longer matched the current control
flow (they fail at the parent commit as well).

## Test Plan

✅ `go test ./...` (only pre-existing runway git-merger environment
failures remain; they fail at the parent commit too) ✅ `make fmt` ✅
`make lint` ✅ `make gazelle` ✅ `make mocks` ✅ storage + orchestrator
integration suites via `make integration-test` targets.


## Stack
1. @ #525
1. #527
@behinddwalls
behinddwalls changed the base branch from preetam/partition-by-queue to main August 6, 2026 16:30
@behinddwalls
behinddwalls marked this pull request as ready for review August 6, 2026 16:31
@behinddwalls
behinddwalls requested review from a team and sbalabanov as code owners August 6, 2026 16:31
@behinddwalls
behinddwalls force-pushed the preetam/queue-on-payloads branch from 7756304 to 229a29a Compare August 6, 2026 16:39
@behinddwalls
behinddwalls changed the base branch from main to preetam/partition-by-queue August 6, 2026 16:39
@behinddwalls
behinddwalls changed the base branch from preetam/partition-by-queue to main August 6, 2026 16:41
## Summary

### Why?

Consumers today learn a message's queue only by loading the referenced entity from storage, and entity IDs are opaque — nothing may parse a queue out of an ID prefix. For storage to become queue-scoped (resolved per queue like every other extension), every consumer must hold the queue before its first storage read, so the queue has to ride on the wire explicitly.

### What?

The internal payload types (`RequestID`, `BatchID`, `BuildID`, `CancelRequest`) gain a `queue` field; the change is additive JSON, and payloads written before the field existed decode with an empty queue. Publishers stamp it from the entity they already hold: start, mergeconflictsignal, batch, speculate, mergesignal, build, buildsignal, and the orchestrator cancel controller; the gateway cancel controller stamps the authoritative queue from the stored request summary, overriding caller input. Consumers guard that a non-empty payload queue matches the loaded entity's queue and reject mismatches as malformed (non-retryable, straight to DLQ) — the guard becomes the routing input once storage is queue-resolved. Buildsignal's re-publish to speculate now partitions by the batch's queue instead of the inherited batch-ID partition key, matching every other speculate publisher and restoring the per-queue serial-processing guarantee that stage relies on.

## Test Plan

✅ `go test ./...` ✅ `make fmt` ✅ `make lint` — new unit tests cover the queue stamp on published payloads and the mismatch rejection guard.
@behinddwalls
behinddwalls force-pushed the preetam/queue-on-payloads branch from 229a29a to 71c44ed Compare August 6, 2026 16:46
@behinddwalls
behinddwalls marked this pull request as draft August 6, 2026 16:46
@behinddwalls
behinddwalls marked this pull request as ready for review August 6, 2026 16:52
@behinddwalls
behinddwalls marked this pull request as draft August 6, 2026 16:56
@behinddwalls
behinddwalls marked this pull request as ready for review August 6, 2026 17:00
Comment thread submitqueue/entity/batch.go
@behinddwalls
behinddwalls merged commit 7d25360 into main Aug 6, 2026
8 of 27 checks passed
@behinddwalls
behinddwalls deleted the preetam/queue-on-payloads branch August 6, 2026 20:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants