Caching in message validation

[Stebalien]

• Drop equivocations before validation because they cannot be useful. Unfortunately, we'd need access to equivocation state.
• Potentially cache valid/invalid messages. Unclear how useful this is, it only helps with rebroadcast.
• Cache merkle-tree hashes. not an issue in practice
• Split pubsub message IDs into "validation IDs" and "propagation IDs"
  • At the validation layer, use the message hash to deduplicate (make sure to make this work with rebroadcast).
  • After successful validation, identify the message based on the hash of the gpbft value.
  • Only propagate messages with unique GPBFT values.

[masih]

Drop equivocations before validation because they cannot be useful.

I think gpbft should do this as an initial check prior to validation.

Potentially cache valid/invalid messages. Unclear how useful this is, it only helps with rebroadcast.

Agreed. We should measure time spent on validation on the mainnet before actioning this bad boy.

[rjan90]

Agreed. We should measure time spent on validation on the mainnet before actioning this bad boy.

@masih Do we have some metrics to see if this is worth prioritising before mainnet launch?

[masih]

Potentially cache valid/invalid messages. Unclear how useful this is, it only helps with rebroadcast.

Agreed. We should measure time spent on validation on the mainnet before actioning this bad boy.

Do we have some metrics to see if this is worth prioritising before mainnet launch?

Yes. This has been measured, and caching already implemented.

[masih]

@rjan90 In terms of ticket management, I recommend breaking this issue up into multiple ones captured in this repo, plus an additional issue at libp2p pubsub repo for the last issue (the f3 part would be integrating with the right hooks once the actual feature is implemented in pubusub).

[Stebalien]

Broken into:

• Drop equivocations before validations in pubsub #602
• Make it possible to coalesce distinct pubsub messages #601