feat: implement gossipsub partial messages extension #577

jxs · 2025-09-16T18:47:13Z

Description

This is a draft implementation of partial messages for gossipsub following the spec PR and based on the Go implementation. Still WIP but should give a good idea of the direction we're heading.

…sipsub-partial-messages

MarcoPolo · 2025-09-16T21:33:16Z

protocols/gossipsub/src/behaviour.rs

+        {
+            // Return err if trying to publish the same partial message state we currently have.
+            if existing.available_parts() == partial_message.available_parts() {
+                return Err(PublishError::Duplicate);


I don't think this is correct.

Imagine you have parts 1,2,3.

You tell your peers about those parts.

A peer comes back and says I want part 2.

You republish with the same parts in order to respond.

You get this error and fail to respond.

thanks for explaining Marco, updated as we spoke

MarcoPolo · 2025-09-16T21:33:54Z

protocols/gossipsub/src/behaviour.rs

    data_transform: D,

+    /// Partial messages received.
+    partial_messages: HashMap<TopicHash, HashMap<Vec<u8>, P>>,


Do you need to store P here? I think it's better if P is owned solely by the application.

yup, you are right, thanks Marco!

MarcoPolo · 2025-09-16T21:35:51Z

protocols/gossipsub/src/types.rs

+pub(crate) struct PartialData {
+    pub(crate) ihave: Vec<u8>,
+    pub(crate) iwant: Vec<u8>,
+    pub(crate) message: Vec<u8>,


Is it useful to store the message here? It seems like wasted space, you only use it to check if the peer is sending you a duplicate.

Might be simpler to let the application handle dupes.

thanks Marco, I only left wanted and has wanted to avoid sending the same message and has to avoid notifying the application layer of duplicates.
Thanks!

Pull-Request: libp2p#6164.

and create a paralel list for the partial_only topics.

Pull-Request: libp2p#6171.

Pull-Request: libp2p#6172.

Noticed two methods which duplicated each other logic precisely, chose the one with more clear name, composed and improved the doc a tiny bit. Pull-Request: libp2p#6173.

This is the up-streaming of sigp#570 which has been beeing used by https://github.com/sigp/lighthouse/ for some weeks now: This started with an attempt to solve libp2p#5751 using the previous internal async-channel. After multiple ideas were discussed off band, replacing the async-channel with an internal more tailored priority queue seemed inevitable. This priority queue allows us to implement the cancellation of in flight IDONTWANT's very cleanly with the `remove_data_messages` function. Clearing the stale messages likewise becomes simpler as we also make use of `remove_data_messages` . Pull-Request: libp2p#6175.

dknopik

Some thoughts and some nitpicks.

dknopik · 2025-10-09T14:23:08Z

protocols/gossipsub/src/types.rs

 #[derive(Debug)]
 pub enum RpcOut {
-    /// Publish a Gossipsub message on network.`timeout` limits the duration the message
+    /// PublishV a Gossipsub message on network.`timeout` limits the duration the message


Accidental change

thanks, updated!

dknopik · 2025-10-09T14:25:06Z

protocols/gossipsub/src/partial.rs

+
+    /// Returns metadata describing which parts of the message are available and which parts we want.
+    ///
+    /// This metadata is application-defined and should encode information about


Partial sentence

thanks Daniel, updated

dknopik · 2025-10-09T14:34:14Z

protocols/gossipsub/src/behaviour.rs

+    // If partial set, filter out peers who only want partial messages for the topic.
+    fn get_publish_peers(&mut self, topic_hash: &TopicHash, partial: bool) -> HashSet<PeerId> {


The fact that partial being true causes the filtering is confusing, as this is set to true on non-partial messages and vice-versa. I'd either call this filter_partial or invert the condition.

yup, agreed and updated. Thanks Daniel!

dknopik · 2025-10-09T14:44:55Z

protocols/gossipsub/src/partial.rs

+    /// - Optional remaining metadata if more parts are still available after this one
+    fn partial_message_bytes_from_metadata(
+        &self,
+        metadata: Option<impl AsRef<[u8]>>,


It is unclear to me what the expected behaviour is if metadata is None. I guess usually send everything we have to the peer?

For Ethereum this is the case where we still haven't received a peer's metadata and could maybe eager push data to them. There are, at a high level, 3 options:

Don't return anything.

Return everything (probably not what we want to do)

Return cells we didn't have locally (cells that we got from the network, not getBlobs).

dknopik · 2025-10-09T14:47:06Z

protocols/gossipsub/src/partial.rs

+    fn partial_message_bytes_from_metadata(
+        &self,
+        metadata: Option<impl AsRef<[u8]>>,
+    ) -> Result<(impl AsRef<[u8]>, Option<impl AsRef<[u8]>>), PartialMessageError>;


If I understand correctly, if we have nothing to send to the peer, we return Ok((vec![], metadata)). This seems unintuitive and potentially inefficient to me, as we have to clone metadata and publish_partial has to compare it to the previous value. Maybe change it to allow returning Ok(None) to signal this?

dknopik · 2025-10-09T14:51:25Z

protocols/gossipsub/src/behaviour.rs

-                    timeout: Delay::new(self.config.publish_queue_duration()),
+                RpcOut::PartialMessage {
+                    message: message_data,
+                    metadata: partial_message.parts_metadata().as_ref().to_vec(),


Maybe get partial_message.parts_metadata() before the loop to avoid useless re-encoding (in case the implementation encodes ad-hoc)?

yeah nice catch, before each peer had its own metadata so this was not possible. Updated!

dknopik · 2025-10-09T15:16:34Z

protocols/gossipsub/src/partial.rs

+    ///
+    /// Returns `Ok(())` if the data was successfully integrated, or `Err`,
+    /// if the data was invalid or couldn't be processed.
+    fn extend_from_encoded_partial_message(


This is unused as far as I can tell. Yes, something like this is likely needed by the application, but as the behaviour does not need to access this, we can leave the exact interface up to the application (as maybe some external info is needed to extend the partial message).

yup, good point. Removed it

dknopik · 2025-10-10T11:27:00Z

protocols/gossipsub/src/behaviour.rs

        self.leave(&topic_hash);
+        #[cfg(feature = "partial_messages")]
+        {
+            self.partial_only_topics.insert(topic_hash.clone());


I think this should be remove?

yup, thanks Daniel updated!

dknopik · 2025-10-10T14:55:44Z

protocols/gossipsub/src/types.rs

+pub(crate) struct PartialData {
+    /// The current peer partial metadata.
+    pub(crate) metadata: Option<Vec<u8>>,
+    /// The remaining heartbeats for this message to be deleted.


Nevermind, we can represent that we sent all data with None.

MarcoPolo · 2025-10-10T20:21:15Z

My review comments as a patch: https://notes.marcopolo.io/view/d804e35e05fa79cfc1ed38e920aa01f1

Import review commit locally:

curl https://notes.marcopolo.io/raw/d804e35e05fa79cfc1ed38e920aa01f1 | git am

Feel free to respond inline with a new patch or quote relevant sections here and respond.

(I'm experimenting with alternative code-review processes, this is a rough prototype of reviews as separate commits. Feedback welcome)

allow updating local data with remote data with monotonicity. I.e if we receive an oudated partial `Metadata::update()` should not override previous data.

dknopik · 2025-10-14T08:36:04Z

protocols/gossipsub/src/partial.rs

+    fn as_slice(&self) -> &[u8];
+    /// try to Update the `Metadata` with the remote data,
+    /// return true if it was updated.
+    fn update(&self, data: &[u8]) -> bool;


what do you think about -> Result<bool, PartialMessageError>?

yup, makes sense. Thanks Daniel

@jxs This also need to take a &mut self

dknopik · 2025-10-14T08:54:02Z

protocols/gossipsub/src/partial.rs

 /// 5. Received partial data is integrated using `extend_from_encoded_partial_message()`
 /// 6. The `group_id()` ties all parts of the same logical message together
 pub trait Partial {
+    type Metadata: Metadata;


should parts_metadata return this?

After discussion, I think the way to go is to remove the assoc type and keep parts_metadata the way it is for now :)

Pull-Request: libp2p#6180.

dknopik · 2025-10-15T09:19:43Z

protocols/gossipsub/src/behaviour.rs

+    pub fn publish_partial<P: Partial>(
+        &mut self,
+        topic: impl Into<TopicHash>,
+        partial_message: P,


…-partial-messages

jxs added 8 commits September 9, 2025 10:48

feat(gossipsub): implement gossipsub 1.3

6313068

implement test extesion

53cc7e8

Add PartialMessage trait implementation

4249ccb

implement sending and receiving partial messages at te protocol level

9aa7ead

add partial_publish method

bc45d3e

update trait extend method to receive slice instead

8b1c086

implement Partial message handling and publishing at the behaviour level

b1b8dbe

Merge branch 'sigp-gossipsub' of github.com:sigp/rust-libp2p into gos…

e7250f5

…sipsub-partial-messages

MarcoPolo reviewed Sep 16, 2025

View reviewed changes

dependabot bot and others added 2 commits September 18, 2025 08:11

deps: bump Swatinem/rust-cache from 2.8.0 to 2.8.1

b433556

Pull-Request: libp2p#6164.

address Marco's review

6370514

jxs force-pushed the gossipsub-partial-messages branch 5 times, most recently from cb0e925 to e6f1ae4 Compare September 26, 2025 11:35

jxs added 3 commits September 26, 2025 15:23

feature gate partial messages

909a68f

remove PartialIDONWANT from proto to follow spec

046ad47

update trait return types and remove non used impl

69c2d95

jxs force-pushed the gossipsub-partial-messages branch from e6f1ae4 to 69c2d95 Compare September 26, 2025 14:24

jxs and others added 10 commits September 26, 2025 15:29

fixup! feature gate partial messages

45c2bc1

update to spec changes

85b92f3

update extension message priority to highest

b6478d5

add missing proto field SubOpts partial

5e98f1a

implement subscription options partial

f77490c

publish only full messages to partial false peers

1bfbc55

revert to only using TopicHash,

33783bd

and create a paralel list for the partial_only topics.

add partial option to subscribe and unsubscribe methods

438a297

chore(ci): remove usage of self-hosted runners (libp2p#6170)

e1929e0

chore: Address latest clippy lints

9d2eb00

Pull-Request: libp2p#6171.

jxs and others added 5 commits October 2, 2025 12:29

chore(ci): address cargo deny issues

e7fea6c

Pull-Request: libp2p#6172.

polish PR

6cb5244

push publish method above for smaller diff

bc3d7ba

fix(gossipsub): remove duplicate config builder method

79f097e

Noticed two methods which duplicated each other logic precisely, chose the one with more clear name, composed and improved the doc a tiny bit. Pull-Request: libp2p#6173.

dknopik reviewed Oct 9, 2025

View reviewed changes

This comment was marked as duplicate.

Sign in to view

dknopik reviewed Oct 10, 2025

View reviewed changes

address Daniel's review

4321429

dknopik reviewed Oct 10, 2025

View reviewed changes

enum return value for partial_message_bytes_from_metadata

c3aa3e4

update Partial implementation

88806ad

allow updating local data with remote data with monotonicity. I.e if we receive an oudated partial `Metadata::update()` should not override previous data.

dknopik reviewed Oct 14, 2025

View reviewed changes

jxs added 3 commits October 14, 2025 11:24

address Daniel's review

4ea2e45

address missing review comment

ea73a71

fix(libp2p): delegate metrics feature to gossipsub when enabled

0e52451

Pull-Request: libp2p#6180.

dknopik reviewed Oct 15, 2025

View reviewed changes

jxs and others added 4 commits October 15, 2025 10:24

impl Daniel's suggestions for the public API

c415f37

borrow metadata mutably for metadata.update

3048649

impl penalization for failed partials

1685aae

address Marco review

69cde69

jxs changed the title ~~Gossipsub partial messages~~ feat: implement gossipsub partial messages extension Oct 15, 2025

jxs added 2 commits October 15, 2025 11:56

cargo fmt

19e359d

Merge branch 'master' of github.com:libp2p/rust-libp2p into gossipsub…

22a7b74

…-partial-messages

		// If partial set, filter out peers who only want partial messages for the topic.
		fn get_publish_peers(&mut self, topic_hash: &TopicHash, partial: bool) -> HashSet<PeerId> {

feat: implement gossipsub partial messages extension #577

Are you sure you want to change the base?

feat: implement gossipsub partial messages extension #577

Conversation

jxs commented Sep 16, 2025

Description

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dknopik left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dknopik Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as duplicate.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MarcoPolo commented Oct 10, 2025

Uh oh!

dknopik Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dknopik Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

dknopik Oct 9, 2025 •

edited

Loading

dknopik Oct 14, 2025 •

edited

Loading

dknopik Oct 14, 2025 •

edited

Loading