Skip to content

Upload protobuf encoding of snapshot/updates to bulk storage. - #6602

Open
mrdziuban wants to merge 1 commit into
canton-network:mainfrom
obsidiansystems:mrdziuban/dump-protobuf-encoding/6497
Open

Upload protobuf encoding of snapshot/updates to bulk storage.#6602
mrdziuban wants to merge 1 commit into
canton-network:mainfrom
obsidiansystems:mrdziuban/dump-protobuf-encoding/6497

Conversation

@mrdziuban

Copy link
Copy Markdown
Contributor

Fixes #6497

This updates SingleAcsSnapshotBulkStorage.getSource and UpdateHistorySegmentBulkStorage.getSource to upload both the API and protobuf encoding of the relevant data to bulk storage.

After incrementing the contracts/updates count metrics, each class:

  • Fans out for both encodings
  • For each encoding, encodes and uploads the data, then increments object count metrics
  • Fans back in so the resulting Source contains the keys of all uploaded objects for both encodings

This Pekko logic is handled by a new generic MultiEncodingBulkStorageFlow, which uses Pekko's Broadcast and Merge to handle the fan-out and fan-in behavior.

Pull Request Checklist

Cluster Testing

  • If a cluster test is required, comment /cluster_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.
  • If an upgrade test is required, comment /upgrade_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.
  • If a hard-migration test is required (from the latest release), comment /hdm_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.
  • If a logical synchronizer upgrade test is required (from canton-3.5), comment /lsu_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.

PR Guidelines

  • Include any change that might be observable by our partners or affect their deployment in the release notes.
  • Specify fixed issues with Fixes #n, and mention issues worked on using #n
  • Include a screenshot for frontend-related PRs - see README or use your favorite screenshot tool

Merge Guidelines

  • Make the git commit message look sensible when squash-merging on GitHub (most likely: just copy your PR description).

@mrdziuban

Copy link
Copy Markdown
Contributor Author

Note I haven't yet tackled this point from #6497

  • Add internal APIs to read the protobuf encoded objects (could wait until we actually need those when we implement DB backfilling)

Let me know if you'd like to see those new APIs be a part of this PR, or would prefer to hold off on them. If I should include them, would it be okay to update the existing /v0/history/bulk/acs and /v0/history/bulk/updates endpoints to add an encoding parameter, or would you prefer to see new endpoints specific to the protobuf encoded objects?

Fixes canton-network#6497.

This updates `SingleAcsSnapshotBulkStorage.getSource` and `UpdateHistorySegmentBulkStorage.getSource` to upload both the API and protobuf encoding of the relevant data to bulk storage.

After incrementing the contracts/updates count metrics, each class:

- Fans out for both encodings
- For each encoding, encodes and uploads the data, then increments object count metrics
- Fans back in so the resulting `Source` contains the keys of all uploaded objects for both encodings

This Pekko logic is handled by a new generic `MultiEncodingBulkStorageFlow`, which uses Pekko's `Broadcast` and `Merge` to handle the fan-out and fan-in behavior.

Signed-off-by: Matt Dziuban <mrdziuban@gmail.com>
@mrdziuban
mrdziuban force-pushed the mrdziuban/dump-protobuf-encoding/6497 branch from 7f6f10a to 2a84ec8 Compare July 29, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dump also protobuf encoding to bulk storage

5 participants