Gossipsub: Partial Message Extension

[MarcoPolo]

All the info is in the spec.

[cskiraly]

As mentioned in our call and prior discussions, this is what we've described in the past as bitmap-based IHAVE/IWANT.
You can find prior art on this e.g. here: https://ethresear.ch/t/fulldas-towards-massive-scalability-with-32mb-blocks-and-beyond/19529#possible-extensions-13

We have already discussed this a few times as the way forward, but haven't arrived to spec it. Hence, I have also started preparing a draft spec on the same a few weeks ago. You can find my draft here:
https://notes.ethereum.org/jAaSLjGBT92iyiNMoSyKNg

It is a bit unfinished, hence I didn't push it here on the repo yet :-) In general I think it is quite aligned with what you propose here. Let's see how we can merge our views into one generic spec.

As I say in my draft above, a structured message ID is used to identify a segment of a larger message. In another view, we can see these as messages in a larger batch of messages. One can easily see that the two are almost exactly the same.

[cskiraly]

Partial message validation is a crucial aspect to this, in fact we can do fast forwarding in FullDAS because we can validate every single message segment individually.

I would dedicate a section to this in the spec (trying to write it up).

[cskiraly]

@MarcoPolo proposed improvements to the partial message spec here
https://github.com/cskiraly/libp2p-specs/tree/csaba/partial-message

In my version I had both partialIHAVE and partialIWANT.

I think I see your point in focusing on the IWANT part, as it might allow us to delegate all the logic to the upper layer, similar to how you can do partial requests in other protocols where you need pieces only (e.g. HTTP Accept-Ranges for resume, and similar use cases). We still need the other node to send the IHAVE (the normal, full IHAVE, and we still would need the logic in our node not to fetch the whole message, so either a hook before IWANT or some call to say "for this I already have something, please send a PartialIWANT with this extra metadata".

If instead we also define the Partial IHAVE, we can do more. In the FullDAS implementation I'm doing these partial IHAVEs to some extent already, because I'm segmenting the column to cell-level messages. We can also give a bitmap encoding to these, as it was proposed there.

Regarding keeping the metadata opaque, I think it is an interesting idea, but it seems to me it requires a more complex API. At the end of the day we do need a protocol layer that is aware of the underlying segment structure. We can try to have this layering of the structure aware and structure unaware part, but I feel this will lead to inefficiencies at some point.

[MarcoPolo]

Let's see how we can merge our views into one generic spec.

That sounds great to me!

Partial message validation is a crucial aspect to this, in fact we can do fast forwarding in FullDAS because we can validate every single message segment individually.

Agreed.

The other use case is faster verification of complete messages. Imagine the fusaka das use case where we are only missing a single cell, we don't need to verify the full column again, just the missing cell.

On fast forwarding, there's a couple different strategies we could do here, but one that stands out to me follows this simple rule:

• Only eagerly push/forward cells we did not already have locally.

The underlying idea is that, as the mempool works today, if you have a cell locally odds are high your peer has it as well (again from your very helpful post: https://ethresear.ch/t/is-data-available-in-the-el-mempool/22329). As we change the mempool, we'll change this strategy as well.

We still need the other node to send the IHAVE (the normal, full IHAVE, and we still would need the logic in our node not to fetch the whole message, so either a hook before IWANT or some call to say "for this I already have something, please send a PartialIWANT with this extra metadata".

This is hard because we don't have a way to link the GroupID to the full message ID present in the IHAVE. One approach is to go all in with partialIHAVEs and use that tell a peer that we have all of the parts of this group, and never send an IHAVE if a peer supports partial message extensions.

Regarding keeping the metadata opaque, I think it is an interesting idea, but it seems to me it requires a more complex API.

I think keeping the metadata opaque and completely application defined is the correct approach. It lets the application define the optimal encoding of the missing. For the das use case bitmaps are very good, but I could also imagine a use case where ranges are desired, or maybe even something like a rateless IBLT.

I feel this will lead to inefficiencies at some point.

What inefficiencies do you have in mind here?

[pawanjay176]

Could this end up increasing the bandwidth because the full message is already in flight by the time you send all the partial control messages? Similar situation to IDONTWANT slightly increasing the bandwidth instead of significantly reducing it.

[MarcoPolo]

Could this end up increasing the bandwidth because the full message is already in flight by the time you send all the partial control messages? Similar situation to IDONTWANT slightly increasing the bandwidth instead of significantly reducing it.

If implemented poorly, yes. But consider this implementation:

Assume your peer supports the partial message extension. Assume we are talking about the data column sidecar case.

1. You never send a full message.
2. If you have some cells from your mempool (via getBlobs), then you lazy push (send an partial IHAVE)
   2a. If you have all the cells from your mempool, then you lazy push a full partial IHAVE message (all bits set to 1).
3. If you don't have a cell already (via getBlobs), you eagerly push that cell to a peer when you do.

This should be strictly better because:

1. You only send the full message if you didn't already have any of the cells.
2. If you have some cells, you will never push a full message.

We could even be lazier, and rely on the peer requesting data rather than lazy pushing data (save 1/2 RTT). We can do this because the peer can request the cell without knowing the hash of the cell. Contrast this with a "normal" gossipsub message where you can only request a message once you know the hash of the message.

[sukunrt]

I want to understand more about how this will interact with the normal(non
partial) message flow on the topic.

As I understand from:
https://github.com/ethereum/consensus-specs/pull/4558/files we want to disable
normal gossipsub for peers that support partial message extension and rely
completely on partial messages to reconstruct the columns. However we want to
keep the old (non partial) system for peers that don't support partial messages.

If this is the case, implementation wise would it be better to make this a
topic level property where only peers with partial message extension can
participate on a partial message specific topic.

[sukunrt]

How is this used?

[MarcoPolo]

It's used by Gossipsub to keep track of the remaining wants for a peer. Imagine this flow:

1. you have parts 1,3
2. peer wants parts 1,2
3. You give the peer part 1.
4. Later, you receive part 2.
5. You should send the peer only part 2.

[MarcoPolo]

Suggestion for text to make this clearer?

[sukunrt]

I'm confused what's handled by gossipsub and what's handled by the application. I assumed this would be handled by the application. As also suggested by my understanding here: #685 (comment)

[MarcoPolo]

I rephrased this, is it clearer?

[MarcoPolo]

If this is the case, implementation wise would it be better to make this a topic level property where only peers with partial message extension can participate on a partial message specific topic.

Currently, in the Go implementation, applications themselves decide if a topic should support partial messages by setting a topic option.

Thinking about this again, maybe it would be better to have an explicit flag in the SubOpts message that requests the receiver to use partial messages.

[sukunrt]

Should we add some suggestions here for implementations. Maybe they should go in: https://github.com/ethereum/consensus-specs/pull/4558/files

Some implementation guidelines, similar to Application Interface, would be helpful:

1. When to send PartialIHave on receiving a piece
2. When to respond to peers PartialIWant.

[MarcoPolo]

Let's keep this thread open as we get more experience with this.