Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase state efficiency for submissions #179

Closed
wants to merge 1 commit into from

Conversation

webmaster128
Copy link
Collaborator

For each submission

  • save 2 bytes for the order index (u32 -> u16)
  • save 10 bytes for the prefix "submissions" -> "s"
  • save 15 bytes for the prefix "submissions_order" -> "so"
  • save 23 bytes in key by canonicalizing address
  • save 23 bytes in value by canonicalizing address

73 bytes/submission = 730 bytes/round = 1460 bytes per minute = 2 MB bytes per day of submissions.

Not sure if it's worth it. Will think about this again.

@kaisbaccour
Copy link
Contributor

kaisbaccour commented Mar 14, 2023

It doesn't decrease code readability by much so I think it should be okay.
But the ultimate scaling would come from a proper lifecycle of data with well thought retention times. So aggregating old data into a very compact form. or maybe just archive it offchain. and delete the rest.

Comment on lines +70 to +78
pub const SUBMISSIONS: Map<(u64, &[u8]), StoredSubmission> = Map::new("s");

/// A map from (round, index) to bot address. This is used when
/// sorted submissions are needed.
///
/// The `index` values are 0-based. So the `n`th submission has index
/// n-1 here as well as in the response array in `SubmissionsResponse`.
/// An entry of this map looks like (round, 1) => drand_bot_addr ; Second fastest bot
pub const SUBMISSIONS_ORDER: Map<(u64, u32), Addr> = Map::new("submissions_order");
pub const SUBMISSIONS_ORDER: Map<(u64, u16), CanonicalAddr> = Map::new("so");
Copy link

@jhernandezb jhernandezb Mar 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you think it could have overlapping iterator issues ? s and so, perhaps add an extra letter to the first map or different letter.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If that is the case I'd consider it a bug in storage-plus. It's possible, not sure. It should be length-prefixed such that this cannot happen.

In general I only want to merge minor changes from this PR. I'm not convinced going to binary addresses is the right way.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right I forgot about storage-plus.

@webmaster128
Copy link
Collaborator Author

For now I'm more confident with #180. Then we should look into proper cleanup logic: #154

@webmaster128 webmaster128 deleted the submissions-state-efficiency branch March 14, 2023 22:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants