New Block structure #3101

deuszx · 2025-01-08T12:00:49Z

Motivation

Verifying Linera (micro)chains with a light client requires following the chain's blocks and subsequently confirming whether an event, a message, state change, were part of that block. Our current block definition - ExecutedBlock type - was not made for this purpose:

there is no distinction between block's "metadata" and the "body" - which would allow for light verification of the blocks.
the events/messages/operations are all layed out, flat, in the BlockExecutionOutcome w/o a way to easily verify whether a particular event is part of that set other than sending the whole vector.

Proposal

In this PR we introduce the new block structures. The new Block type consists of two elements:

(block) header
(block) body.

BlockHeader is a succint representation of the chain's new block. It contains metadata that suffices to track the chain's progress:

pub struct BlockHeader {
    /// The block version.
    pub version: u8, // TODO: More granular versioning. #3078
    /// The chain to which this block belongs.
    pub chain_id: ChainId,
    /// The number identifying the current configuration.
    pub epoch: Epoch,
    /// The block height.
    pub height: BlockHeight,
    /// The timestamp when this block was created.
    pub timestamp: Timestamp,
    /// The hash of the chain's execution state after this block.
    pub state_hash: CryptoHash,
    /// Certified hash of the previous block in the
    /// chain, if any.
    pub previous_block_hash: Option<CryptoHash>,
    /// The user signing for the operations in the block and paying for their execution
    /// fees. If set, this must be the `owner` in the block proposal. `None` means that
    /// the default account of the chain is used. This value is also used as recipient of
    /// potential refunds for the message grants created by the operations.
    pub authenticated_signer: Option<Owner>,

    // Inputs to the block, chosen by the block proposer.
    /// Cryptographic hash of all the incoming bundles in the block.
    pub bundles_hash: CryptoHash,
    /// Cryptographic hash of all the operations in the block.
    pub operations_hash: CryptoHash,

    // Outcome of the block execution.
    /// Cryptographic hash of all the messages in the block.
    pub messages_hash: CryptoHash,
    /// Cryptographic hash of all the oracle responses in the block.
    pub oracle_responses_hash: CryptoHash,
    /// Cryptographic hash of all the events in the block.
    pub events_hash: CryptoHash,
}

BlockBody holds the data that is not necessary for (light) verification but was either an input to the block's execution (incoming bundles, operations) or its output (oracle responses, messages sent, events emitted). Each vector is deterministically hashed (into CryptoHash) and the result of that hashing is put into the BlockHeader. This way it's possible to verify inclusion of any specific element w/o having to sent all of the body.

/// The body of a block containing all the data included in the block.
#[derive(Debug, PartialEq, Eq, Hash, Clone, Serialize, Deserialize, SimpleObject)]
pub struct BlockBody {
    /// A selection of incoming messages to be executed first. Successive messages of same
    /// sender and height are grouped together for conciseness.
    pub incoming_bundles: Vec<IncomingBundle>,
    /// The operations to execute.
    pub operations: Vec<Operation>,
    /// The list of outgoing messages for each transaction.
    pub messages: Vec<Vec<OutgoingMessage>>,
    /// The record of oracle responses for each transaction.
    pub oracle_responses: Vec<Vec<OracleResponse>>,
    /// The list of events produced by each transaction.
    pub events: Vec<Vec<EventRecord>>,
}

Test Plan

All of the tests have been updated to work with the new types.

Release Plan

These changes follow the usual release cycle. Although we may want to defer this update since this change is not backwards compatible.

Links

linera-chain/src/manager.rs

deuszx · 2025-01-09T14:11:48Z

linera-core/src/client/mod.rs

-                if certificate.executed_block().block
-                    == confirmed_value.inner().executed_block().block =>
+                if certificate.block() == confirmed_value.inner().block() =>
+            // TODO: double check this. Previously we were comparing the Proposal only.


TODO: Double-check. Same here as above.

linera-chain/src/chain.rs

linera-base/src/crypto.rs

linera-core/src/unit_tests/worker_tests.rs

linera-base/src/crypto.rs

ma2bd · 2025-01-10T17:16:21Z

linera-chain/src/certificate/confirmed.rs

    }

    /// Returns whether this value contains the message with the specified ID.
    pub fn has_message(&self, message_id: &MessageId) -> bool {
-        self.executed_block().message_by_id(message_id).is_some()
+        self.block().message_by_id(message_id).is_some()


Would it make sense the handle the renaming executed_block -> block first in a separate PR?

Actually, I didn't get rid of ExecutedBlock. I tried but the Proposal/BlockExecutionOutcome is used in a couple of places (we execute before proposing, we execute when we receive a proposal, we re-execute when we receive a confirmed certificate, etc.) so I kept the ExecutedBlock as an internal data structure that caries the result of execution.

This particular change that you commented on is a follow-up to changing the ConfirmedBlock(ExecutedBlock) to ConfirmedBlock(Block) so it's not a rename only, it's actually a new Block type inside.

When (de)serializing a Block, don't (de)serialize the hashes from the BlockHeader but rather compute them from the elements of the BlockBody. This makes sure that we never simply _trust_ the values that are being sent and rather compute them ourselves.

afck · 2025-01-13T14:09:33Z

linera-chain/src/data_types.rs

@@ -395,16 +399,16 @@ impl OutgoingMessage {
    }
 }

-/// A [`Block`], together with the outcome from its execution.
+/// A [`Proposal`], together with the outcome from its execution.
 #[derive(Debug, PartialEq, Eq, Hash, Clone, Serialize, Deserialize, SimpleObject)]
 pub struct ExecutedBlock {


Maybe ProposalWithOutcome. Or even inline it, if it's not used in many places anymore?

afck · 2025-01-13T14:10:27Z

linera-chain/src/data_types.rs

@@ -76,7 +76,7 @@ pub struct Block {
    pub previous_block_hash: Option<CryptoHash>,
 }

-impl Block {
+impl Proposal {


Now a BlockProposal contains a ProposalContent which contains a Proposal.

Maybe call this type ProposedBlock?

I didn't call it ProposedBlock on purpose b/c we have the BlockProposal which sounds basically equivalent that is the encompassing type. It seemed weird that BlockProposal contains ProposedBlock.

But isn't a BlockProposal containing a Proposal even worse?

Maybe BlockInput? BlockTemplate? BlockPlan? 😕

linera-base/src/crypto.rs

linera-chain/src/block.rs

linera-chain/src/chain.rs

linera-chain/src/block.rs

linera-chain/src/manager.rs

afck · 2025-01-15T12:06:45Z

linera-chain/src/test.rs

-        authenticated_signer: parent_block.authenticated_signer,
-        timestamp: parent_block.timestamp,
+        height: parent_header.height.try_add_one().unwrap(),
+        authenticated_signer: parent_header.authenticated_signer,


(Weird! What if this is proposed by someone else? Not new in this PR, of course.)

linera-chain/src/unit_tests/chain_tests.rs

afck

No blockers from my side, but:

Can we remove ExecutedBlock now? (Maybe in another PR.)
One test is still failing.
Since this is a very central type, it would be great if we could get a second review.

deuszx · 2025-01-17T15:34:43Z

Can we remove ExecutedBlock now? (Maybe in another PR.)

Discussed offline - and decided not to do that. ExecutedBlock is still being used as a return type from execution. it's also more useful as the internal, intermediate type b/c it retains the original proposal <> outcome distinction (Block is the end type) and that is used in client (due to pre-execution before sending the proposal) and on the server to differentiate between initial proposal and a retry.

One test is still failing.

I think I fixed that now.

Since this is a very central type, it would be great if we could get a second review.

I agree, let's wait.

## Motivation The discussion #3101 (comment) was never resolved; a proposal has a content, which has a proposal. ## Proposal Rename `Proposal` to `ProposedBlock`, to distinguish it from the outer type. (I also added some destructuring in `Block::new`, because I ran into a naming conflict and this way the compiler will warn us if we add any fields and forget them there.) ## Test Plan Only renaming. ## Release Plan - Nothing to do / These changes follow the usual release cycle. ## Links - Related to #3101 - [reviewer checklist](https://github.com/linera-io/linera-protocol/blob/main/CONTRIBUTING.md#reviewer-checklist)

Rename Block to Proposal

bc152a7

deuszx force-pushed the new-block-structure branch from 96cbde5 to 6ab368e Compare January 8, 2025 14:30

deuszx added 2 commits January 8, 2025 18:51

New Block structures

a6080da

Add a newtype struct for wrapping Vec<CryptoHash>

facb07d

deuszx force-pushed the new-block-structure branch from 6ab368e to facb07d Compare January 9, 2025 12:01

deuszx added 4 commits January 9, 2025 13:21

fix linera-indexer/examples test

c3ccffa

Fix linera-explorer tests

a61b959

Fix hex_game example

5204ea3

Fix cargo clippy

1ab41d6