feat(submitqueue): add immutable receipt lookup

Summary:
First step toward keeping one full request projection in request_summary, with a small time-based lookup instead of duplicating lifecycle data for List. All required response data and receipt time are already captured.

### Changes

- Add request_receipt with only (queue, received_at_ms, request_id), its immutable storage contract, and gateway wiring.
- Ensure receipt keys after public projection writes, including unchanged/stale-log retries. Keep accepting receipts hidden and surface partial-write failures for existing retry handling.
- Keep List reads and the existing queue-summary projection writes in place for a safe cutover.
- Add storage, materializer, and integration coverage; document the transition in the RFC.

### Migration plan

1. Deploy the additive table before rolling out these writes. This PR populates receipt keys alongside the existing projection; it does not switch List reads.
2. Upgrade all writers, then backfill existing List membership from request_summary_by_queue keys. Validate keys against public authoritative summaries, exclude accepting receipts, and verify coverage before cutover.
3. In a follow-up, switch List to scan receipt keys and point-read request_summary. Preserve the API, time bounds, ordering, and page tokens; check page latency and keep old projection writes during the rollback window.
4. Once cutover is stable, stop duplicate projection writes, remove RequestQueueSummary and its store/mapper, and retire request_summary_by_queue after the rollback window. The final model stores one full projection plus immutable lookup keys.

Backfill tooling, read cutover, and duplicate-data removal are follow-up work. This preserves currently listed history without reconstructing requests predating the original read-model rollout.

Test Plan:
Before the internal rollout, provision the new table and confirm newly public requests produce matching receipt keys while List continues serving the old projection. Existing automated tests cover retries, concurrency, hidden receipts, and pagination.

Revert Plan:
Revert the writer rollout; List still uses the existing projection. Leave the additive table in place until no writers depend on it.

API Changes:
No RPC/proto changes. Gateway storage implementations must provide the new RequestReceiptStore accessor.

Author: mnoah1

doc/rfc/submitqueue/status-list-api.md

@@ -77,7 +77,7 @@ The transport representation of these errors follows the gateway-wide RPC error
 
 ## Gateway-Owned Read Model
 
-The gateway owns the append-only request log and three new logical read models. The orchestrator's request and change stores are pipeline working state with different retention semantics, so neither API reads them.
+The gateway owns the append-only request log, materialized request summaries, and lookup stores. The orchestrator's request and change stores are pipeline working state with different retention semantics, so neither API reads them.
 
 ### Request Summary by Sqid
 
@@ -91,7 +91,7 @@ The key supports authoritative lookup and conditional status updates for one req
 
 The queue projection is logically keyed by `(queue, received_at_ms, sqid)` and must support a bounded descending scan over `(received_at_ms, sqid)`. A backend may satisfy that contract with a reverse range scan or by descending-encoding the ordered key components; cursors always carry the original receipt time and sqid values.
 
-The row duplicates the complete `List` response deliberately. A page is served by one bounded range scan rather than one follow-up authoritative-summary read per result. Status updates propagate from the authoritative sqid summary to this projection.
+The current `request_summary_by_queue` row duplicates the complete `List` response so a page is served by one bounded range scan. Status updates still propagate to this projection during the receipt-lookup transition below; it is not the target storage model.
 
 The logical key covers the `List` queue predicate, receipt-time range, newest-first ordering, and complete keyset cursor in one bounded scan.
 
@@ -162,6 +162,20 @@ There is intentionally no backfill from orchestrator working state or historical
 
 The request-summary RPCs replace the former polymorphic Status shape with explicit ID and change-URI lookups. This is a source-breaking API change. The List RPC is additive.
 
+### Receipt-Lookup Transition
+
+The target model keeps the full projection only in `request_summary`. An immutable `request_receipt` lookup contains only `(queue, received_at_ms, request_id)` as its primary key. List will scan these keys and point-read the corresponding authoritative summaries, preserving the existing response, required time window, string-ID tie ordering, and page tokens. Receipt time and all response fields are already captured; no additional data collection is needed.
+
+The first phase adds the lookup and its writes, without changing List reads or removing the old projection. After materializing a public summary and completing the existing URI/queue projection writes, the materializer ensures the receipt key exists, including on unchanged or stale-log retries. Hidden `accepting` receipts are excluded. A failed lookup write is returned through the existing projection-failure path.
+
+Deploy the additive table before enabling these writes. To finish the transition:
+
+1. Upgrade all writers, backfill existing List membership from the old table's keys, validate each key against its public authoritative summary, and verify lookup coverage.
+2. Switch List to the receipt lookup plus authoritative summary reads, keeping old projection writes during the rollback window. This trades one page scan for one scan plus a point read per visible result; verify page latency before cutover.
+3. Stop duplicate projection writes, remove `RequestQueueSummary` and its store/mapper, and retire `request_summary_by_queue` after the rollback window. Only the small immutable lookup key remains duplicated.
+
+This backfill preserves existing List history; it does not reconstruct requests predating the original read-model rollout. Backfill tooling, read cutover, and removal are follow-up work.
+
 ## Rejected Alternatives
 
 ### Put Request Context in Every Log Entry

submitqueue/entity/BUILD.bazel

@@ -20,6 +20,7 @@ go_library(
         "request_batch.go",
         "request_history.go",
         "request_log.go",
+        "request_receipt.go",
         "request_summary.go",
         "speculation.go",
         "subject.go",

submitqueue/entity/request_receipt.go

@@ -0,0 +1,25 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package entity
+
+// RequestReceipt is an immutable lookup key for a publicly visible request.
+type RequestReceipt struct {
+	// Queue is the queue supplied at receipt.
+	Queue string
+	// ReceivedAtMs is the immutable, positive receipt timestamp in Unix milliseconds.
+	ReceivedAtMs int64
+	// RequestID identifies the request within Queue.
+	RequestID string
+}

submitqueue/extension/storage/BUILD.bazel

@@ -12,6 +12,7 @@ go_library(
         "request_batch_store.go",
         "request_log_store.go",
         "request_queue_summary_store.go",
+        "request_receipt_store.go",
         "request_store.go",
         "request_summary_store.go",
         "request_uri_store.go",

submitqueue/extension/storage/mock/BUILD.bazel

@@ -12,6 +12,7 @@ go_library(
         "request_batch_store_mock.go",
         "request_log_store_mock.go",
         "request_queue_summary_store_mock.go",
+        "request_receipt_store_mock.go",
         "request_store_mock.go",
         "request_summary_store_mock.go",
         "request_uri_store_mock.go",

submitqueue/extension/storage/mock/request_receipt_store_mock.go

@@ -0,0 +1,54 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package storage
+
+//go:generate mockgen -source=request_receipt_store.go -destination=mock/request_receipt_store_mock.go -package=mock
+
+import (
+	"context"
+
+	"github.com/uber/submitqueue/submitqueue/entity"
+)
+
+// RequestReceiptCursor is an exclusive position in descending receipt-time order.
+type RequestReceiptCursor struct {
+	// ReceivedAtMs is the positive receipt timestamp in Unix milliseconds; zero with an empty RequestID starts the range.
+	ReceivedAtMs int64
+	// RequestID breaks timestamp ties in descending string order, not numeric order.
+	RequestID string
+}
+
+// RequestReceiptRange bounds a primary-key scan within the store's queue.
+type RequestReceiptRange struct {
+	// ReceivedAtOrAfterMs is the inclusive lower receipt-time bound in Unix milliseconds.
+	ReceivedAtOrAfterMs int64
+	// ReceivedBeforeMs is the exclusive upper receipt-time bound and must exceed the lower bound.
+	ReceivedBeforeMs int64
+	// Before is the exclusive continuation boundary; its zero value starts the range.
+	Before RequestReceiptCursor
+	// Limit is the positive maximum number of mappings to return.
+	Limit int
+}
+
+// RequestReceiptStore retains immutable receipt mappings in its bound queue.
+type RequestReceiptStore interface {
+	// Create rejects queue mismatches, nonpositive receipt times, and empty request IDs.
+	// An existing composite key returns ErrAlreadyExists.
+	Create(ctx context.Context, receipt entity.RequestReceipt) error
+
+	// List scans the primary-key range in descending (received_at_ms, request_id) order.
+	// It returns at most Limit mappings, an empty slice when none match, and an error for invalid ranges.
+	List(ctx context.Context, bounds RequestReceiptRange) ([]entity.RequestReceipt, error)
+}

submitqueue/extension/storage/request_receipt_store.go

@@ -348,6 +348,7 @@ func TestLand_PublishesToQueue(t *testing.T) {
 	var materializedSummary entity.RequestSummary
 	var persistedMapping entity.RequestURI
 	var persistedQueueSummary entity.RequestQueueSummary
+	var persistedReceipt entity.RequestReceipt
 	var persistedLog entity.RequestLog
 
 	ctrl := gomock.NewController(t)
@@ -359,8 +360,10 @@ func TestLand_PublishesToQueue(t *testing.T) {
 	summaryStore := storagemock.NewMockRequestSummaryStore(ctrl)
 	uriStore := storagemock.NewMockRequestURIStore(ctrl)
 	queueStore := storagemock.NewMockRequestQueueSummaryStore(ctrl)
+	receiptStore := storagemock.NewMockRequestReceiptStore(ctrl)
 	logStore := storagemock.NewMockRequestLogStore(ctrl)
 	store.EXPECT().GetRequestQueueSummaryStore().Return(queueStore).AnyTimes()
+	store.EXPECT().GetRequestReceiptStore().Return(receiptStore).AnyTimes()
 	store.EXPECT().GetRequestSummaryStore().Return(summaryStore).AnyTimes()
 	store.EXPECT().GetRequestLogStore().Return(logStore).AnyTimes()
 	store.EXPECT().GetRequestURIStore().Return(uriStore).AnyTimes()
@@ -418,6 +421,12 @@ func TestLand_PublishesToQueue(t *testing.T) {
 				return nil
 			},
 		),
+		receiptStore.EXPECT().Create(gomock.Any(), gomock.Any()).DoAndReturn(
+			func(_ context.Context, receipt entity.RequestReceipt) error {
+				persistedReceipt = receipt
+				return nil
+			},
+		),
 	)
 
 	controller := NewLandController(zap.NewNop().Sugar(), tally.NoopScope, staticCounterFactory{counter: cnt}, factoryForStorage(ctrl, store), materializer, noopQueueConfigStore(ctrl), registry)
@@ -444,6 +453,7 @@ func TestLand_PublishesToQueue(t *testing.T) {
 		Metadata:          map[string]string{},
 	}, receiptSummary)
 	assert.Positive(t, receiptSummary.ReceivedAtMs)
+	assert.Equal(t, entity.RequestReceipt{Queue: req.Queue, ReceivedAtMs: receiptSummary.ReceivedAtMs, RequestID: result.ID}, persistedReceipt)
 	assert.Equal(t, entity.RequestLog{
 		RequestID:   "123",
 		Queue:       "test-queue",

submitqueue/gateway/controller/land_test.go

@@ -141,8 +141,10 @@ func newLogControllerStore(ctrl *gomock.Controller, insertErr, getErr, updateErr
 	logStore := storagemock.NewMockRequestLogStore(ctrl)
 	summaryStore := storagemock.NewMockRequestSummaryStore(ctrl)
 	queueStore := storagemock.NewMockRequestQueueSummaryStore(ctrl)
+	receiptStore := storagemock.NewMockRequestReceiptStore(ctrl)
 	uriStore := storagemock.NewMockRequestURIStore(ctrl)
 	store.EXPECT().GetRequestQueueSummaryStore().Return(queueStore).AnyTimes()
+	store.EXPECT().GetRequestReceiptStore().Return(receiptStore).AnyTimes()
 	store.EXPECT().GetRequestSummaryStore().Return(summaryStore).AnyTimes()
 	store.EXPECT().GetRequestLogStore().Return(logStore).AnyTimes()
 	store.EXPECT().GetRequestURIStore().Return(uriStore).AnyTimes()
@@ -169,6 +171,9 @@ func newLogControllerStore(ctrl *gomock.Controller, insertErr, getErr, updateErr
 		Status: entity.RequestStatusAccepted, Version: 1, Metadata: map[string]string{},
 	}, nil)
 	queueStore.EXPECT().Update(gomock.Any(), gomock.Any(), int32(1), int32(2)).Return(queueErr)
+	if queueErr == nil {
+		receiptStore.EXPECT().Create(gomock.Any(), entity.RequestReceipt{Queue: "test-queue", ReceivedAtMs: 1, RequestID: "2"}).Return(nil)
+	}
 	return materializer
 }
 

submitqueue/gateway/controller/log/log_test.go

@@ -33,6 +33,7 @@ type controllerStorageFixture struct {
 	storage        *gwstoragemock.MockStorage
 	summaryStore   *storagemock.MockRequestSummaryStore
 	queueStore     *storagemock.MockRequestQueueSummaryStore
+	receiptStore   *storagemock.MockRequestReceiptStore
 	uriStore       *storagemock.MockRequestURIStore
 	logStore       *storagemock.MockRequestLogStore
 	mu             sync.Mutex
@@ -47,12 +48,14 @@ func newControllerStorageFixture(ctrl *gomock.Controller) *controllerStorageFixt
 		storage:        gwstoragemock.NewMockStorage(ctrl),
 		summaryStore:   storagemock.NewMockRequestSummaryStore(ctrl),
 		queueStore:     storagemock.NewMockRequestQueueSummaryStore(ctrl),
+		receiptStore:   storagemock.NewMockRequestReceiptStore(ctrl),
 		uriStore:       storagemock.NewMockRequestURIStore(ctrl),
 		logStore:       storagemock.NewMockRequestLogStore(ctrl),
 		summaries:      make(map[string]entity.RequestSummary),
 		queueSummaries: make(map[string]entity.RequestQueueSummary),
 	}
 	fixture.storage.EXPECT().GetRequestQueueSummaryStore().Return(fixture.queueStore).AnyTimes()
+	fixture.storage.EXPECT().GetRequestReceiptStore().Return(fixture.receiptStore).AnyTimes()
 	fixture.storage.EXPECT().GetRequestSummaryStore().Return(fixture.summaryStore).AnyTimes()
 	fixture.storage.EXPECT().GetRequestLogStore().Return(fixture.logStore).AnyTimes()
 	fixture.storage.EXPECT().GetRequestURIStore().Return(fixture.uriStore).AnyTimes()
@@ -127,6 +130,7 @@ func newControllerStorageFixture(ctrl *gomock.Controller) *controllerStorageFixt
 	}).AnyTimes()
 
 	fixture.uriStore.EXPECT().Create(gomock.Any(), gomock.Any()).Return(nil).AnyTimes()
+	fixture.receiptStore.EXPECT().Create(gomock.Any(), gomock.Any()).Return(nil).AnyTimes()
 	fixture.logStore.EXPECT().Insert(gomock.Any(), gomock.Any()).DoAndReturn(func(_ context.Context, log entity.RequestLog) error {
 		fixture.mu.Lock()
 		defer fixture.mu.Unlock()