New Block structure #3101

deuszx · 2025-01-08T12:00:49Z

Motivation

Verifying Linera (micro)chains with a light client requires following the chain's blocks and subsequently confirming whether an event, a message, state change, were part of that block. Our current block definition - ExecutedBlock type - was not made for this purpose:

there is no distinction between block's "metadata" and the "body" - which would allow for light verification of the blocks.
the events/messages/operations are all layed out, flat, in the BlockExecutionOutcome w/o a way to easily verify whether a particular event is part of that set other than sending the whole vector.

Proposal

In this PR we introduce the new block structures. The new Block type consists of two elements:

(block) header
(block) body.

BlockHeader is a succint representation of the chain's new block. It contains metadata that suffices to track the chain's progress:

pub struct BlockHeader {
    /// The block version.
    pub version: u8, // TODO: More granular versioning. #3078
    /// The chain to which this block belongs.
    pub chain_id: ChainId,
    /// The number identifying the current configuration.
    pub epoch: Epoch,
    /// The block height.
    pub height: BlockHeight,
    /// The timestamp when this block was created.
    pub timestamp: Timestamp,
    /// The hash of the chain's execution state after this block.
    pub state_hash: CryptoHash,
    /// Certified hash of the previous block in the
    /// chain, if any.
    pub previous_block_hash: Option<CryptoHash>,
    /// The user signing for the operations in the block and paying for their execution
    /// fees. If set, this must be the `owner` in the block proposal. `None` means that
    /// the default account of the chain is used. This value is also used as recipient of
    /// potential refunds for the message grants created by the operations.
    pub authenticated_signer: Option<Owner>,

    // Inputs to the block, chosen by the block proposer.
    /// Cryptographic hash of all the incoming bundles in the block.
    pub bundles_hash: CryptoHash,
    /// Cryptographic hash of all the operations in the block.
    pub operations_hash: CryptoHash,

    // Outcome of the block execution.
    /// Cryptographic hash of all the messages in the block.
    pub messages_hash: CryptoHash,
    /// Cryptographic hash of all the oracle responses in the block.
    pub oracle_responses_hash: CryptoHash,
    /// Cryptographic hash of all the events in the block.
    pub events_hash: CryptoHash,
}

BlockBody holds the data that is not necessary for (light) verification but was either an input to the block's execution (incoming bundles, operations) or its output (oracle responses, messages sent, events emitted). Each vector is deterministically hashed (into CryptoHash) and the result of that hashing is put into the BlockHeader. This way it's possible to verify inclusion of any specific element w/o having to sent all of the body.

/// The body of a block containing all the data included in the block.
#[derive(Debug, PartialEq, Eq, Hash, Clone, Serialize, Deserialize, SimpleObject)]
pub struct BlockBody {
    /// A selection of incoming messages to be executed first. Successive messages of same
    /// sender and height are grouped together for conciseness.
    pub incoming_bundles: Vec<IncomingBundle>,
    /// The operations to execute.
    pub operations: Vec<Operation>,
    /// The list of outgoing messages for each transaction.
    pub messages: Vec<Vec<OutgoingMessage>>,
    /// The record of oracle responses for each transaction.
    pub oracle_responses: Vec<Vec<OracleResponse>>,
    /// The list of events produced by each transaction.
    pub events: Vec<Vec<EventRecord>>,
}

Test Plan

All of the tests have been updated to work with the new types.

Release Plan

These changes follow the usual release cycle. Although we may want to defer this update since this change is not backwards compatible.

Links

deuszx · 2025-01-09T13:55:03Z

linera-chain/src/manager.rs

        let new_round = certificate.round;
        if let Some(Vote { value, round, .. }) = &self.confirmed_vote {
-            if value.inner().executed_block().block == *new_block && *round == new_round {
+            if value.inner().block() == new_block && *round == new_round {


TODO: Double-check. Previously we were comparing the Proposal and now we compare the new Block type. It means we also compare the execution results now.

deuszx · 2025-01-09T14:11:48Z

linera-core/src/client/mod.rs

-                if certificate.executed_block().block
-                    == confirmed_value.inner().executed_block().block =>
+                if certificate.block() == confirmed_value.inner().block() =>
+            // TODO: double check this. Previously we were comparing the Proposal only.


TODO: Double-check. Same here as above.

linera-chain/src/chain.rs

deuszx · 2025-01-09T14:25:43Z

linera-chain/src/chain.rs

@@ -1235,5 +1241,5 @@ fn empty_executed_block_size() {
        outcome: crate::data_types::BlockExecutionOutcome::default(),
    };
    let size = bcs::serialized_size(&executed_block).unwrap();
-    assert_eq!(size, EMPTY_EXECUTED_BLOCK_SIZE);
+    assert_eq!(size, EMPTY_BLOCK_SIZE);


TODO: Test for empty block (new) size.

Done in Testy empty Block size.

What's the point of testing a test-only variable? (But as I said I think we should just use this one in production.)

You're right - it doesn't make sense to keep this test for the ExecutedBlock type.

linera-base/src/crypto.rs

linera-core/src/unit_tests/worker_tests.rs

ma2bd · 2025-01-10T17:15:38Z

linera-base/src/crypto.rs

+    where
+        S: serde::ser::Serializer,
+    {
+        serializer.serialize_newtype_struct("CryptoHashVec", &self.0)


Another #[transparent]?

Ah, good point.

Done in dca5da3.

Actually, this won't work, b/c we will start getting the errors that we can call type_name() only on struct or newtype (b/c we'll call Deserialize on the Vec<_> and that will fail).

Also why do we need this in the first place?

When constructing the new Block, we are hashing the contents of the BlockBody - we iterate over elements of incoming_bundles, operations, etc. turning them into CryptoHash-es and then turning resulting Vec<CryptoHash> into a CryptoHash - here. When calling CryptoHash::new(Vec<CryptoHash>) the code would fail b/c calling serde_name::trace_name on a Vec throws. To work around that we have this newtype struct that wraps the Vec<CryptoHash> first.

Wouldn't deriving Serialize and Deserialize (without #[transparent]) do exactly the same as this implementation?

ma2bd · 2025-01-10T17:16:21Z

linera-chain/src/certificate/confirmed.rs

    }

    /// Returns whether this value contains the message with the specified ID.
    pub fn has_message(&self, message_id: &MessageId) -> bool {
-        self.executed_block().message_by_id(message_id).is_some()
+        self.block().message_by_id(message_id).is_some()


Would it make sense the handle the renaming executed_block -> block first in a separate PR?

Actually, I didn't get rid of ExecutedBlock. I tried but the Proposal/BlockExecutionOutcome is used in a couple of places (we execute before proposing, we execute when we receive a proposal, we re-execute when we receive a confirmed certificate, etc.) so I kept the ExecutedBlock as an internal data structure that caries the result of execution.

This particular change that you commented on is a follow-up to changing the ConfirmedBlock(ExecutedBlock) to ConfirmedBlock(Block) so it's not a rename only, it's actually a new Block type inside.

linera-sdk/src/test/block.rs

linera-indexer/example/tests/test.rs

linera-core/src/unit_tests/worker_tests.rs

linera-chain/src/block.rs

When (de)serializing a Block, don't (de)serialize the hashes from the BlockHeader but rather compute them from the elements of the BlockBody. This makes sure that we never simply _trust_ the values that are being sent and rather compute them ourselves.

afck · 2025-01-13T14:09:33Z

linera-chain/src/data_types.rs

@@ -395,16 +399,16 @@ impl OutgoingMessage {
    }
 }

-/// A [`Block`], together with the outcome from its execution.
+/// A [`Proposal`], together with the outcome from its execution.
 #[derive(Debug, PartialEq, Eq, Hash, Clone, Serialize, Deserialize, SimpleObject)]
 pub struct ExecutedBlock {


Maybe ProposalWithOutcome. Or even inline it, if it's not used in many places anymore?

afck · 2025-01-13T14:10:27Z

linera-chain/src/data_types.rs

@@ -76,7 +76,7 @@ pub struct Block {
    pub previous_block_hash: Option<CryptoHash>,
 }

-impl Block {
+impl Proposal {


Now a BlockProposal contains a ProposalContent which contains a Proposal.

Maybe call this type ProposedBlock?

I didn't call it ProposedBlock on purpose b/c we have the BlockProposal which sounds basically equivalent that is the encompassing type. It seemed weird that BlockProposal contains ProposedBlock.

But isn't a BlockProposal containing a Proposal even worse?

Maybe BlockInput? BlockTemplate? BlockPlan? 😕

afck · 2025-01-13T17:00:17Z

linera-base/src/crypto.rs

+    where
+        S: serde::ser::Serializer,
+    {
+        serializer.serialize_newtype_struct("CryptoHashVec", &self.0)


Wouldn't deriving Serialize and Deserialize (without #[transparent]) do exactly the same as this implementation?

afck · 2025-01-13T17:01:14Z

linera-chain/src/block.rs

+
+/// Block defines the atomic unit of growth of the Linera chain.
+/// As part of the block body, contains all the incoming messages
+/// and operations to executed which define state transition of the chain.


Suggested change

/// and operations to executed which define state transition of the chain.

/// and operations to execute which define a state transition of the chain.

afck · 2025-01-13T17:02:20Z

linera-chain/src/block.rs

@@ -192,3 +206,362 @@ pub enum ConversionError {
    #[error("Expected a `TimeoutCertificate` value")]
    Timeout,
 }
+
+/// Block defines the atomic unit of growth of the Linera chain.
+/// As part of the block body, contains all the incoming messages


I think it's common to add a blank line (i.e. new paragraph in markdown; also below).

Suggested change

/// As part of the block body, contains all the incoming messages

///

/// As part of the block body, contains all the incoming messages

afck · 2025-01-13T17:06:56Z

linera-chain/src/chain.rs

@@ -147,9 +148,13 @@ static STATE_HASH_COMPUTATION_LATENCY: LazyLock<HistogramVec> = LazyLock::new(||
    )
 });

-/// The BCS-serialized size of an empty `ExecutedBlock`.
+/// The BCS-serialized size of an empty [`crate::data_types::ExecutedBlock`].


Do we still need to serialize ExtecutedBlocks at all? Can't we always use Block instead?

And this constant is about the block size limit: I'd say that should apply to Block, not to ExecutedBlock?

You're right. We don't need them anymore.

afck · 2025-01-13T17:10:20Z

linera-chain/src/chain.rs

@@ -698,7 +707,7 @@ where
            account: block.authenticated_signer,
        };
        resource_controller
-            .track_executed_block_size(EMPTY_EXECUTED_BLOCK_SIZE)
+            .track_block_size(EMPTY_EXECUTED_BLOCK_SIZE)


I'd rather use EMPTY_BLOCK_SIZE here.

Suggested change

.track_block_size(EMPTY_EXECUTED_BLOCK_SIZE)

.track_block_size(EMPTY_EXECUTED_BLOCK_SIZE)

afck · 2025-01-13T17:11:18Z

linera-chain/src/chain.rs

@@ -1235,5 +1241,5 @@ fn empty_executed_block_size() {
        outcome: crate::data_types::BlockExecutionOutcome::default(),
    };
    let size = bcs::serialized_size(&executed_block).unwrap();
-    assert_eq!(size, EMPTY_EXECUTED_BLOCK_SIZE);
+    assert_eq!(size, EMPTY_BLOCK_SIZE);


What's the point of testing a test-only variable? (But as I said I think we should just use this one in production.)

Rename Block to Proposal

bc152a7

deuszx force-pushed the new-block-structure branch from 96cbde5 to 6ab368e Compare January 8, 2025 14:30

deuszx added 2 commits January 8, 2025 18:51

New Block structures

a6080da

Add a newtype struct for wrapping Vec<CryptoHash>

facb07d

deuszx force-pushed the new-block-structure branch from 6ab368e to facb07d Compare January 9, 2025 12:01

deuszx added 4 commits January 9, 2025 13:21

fix linera-indexer/examples test

c3ccffa

Fix linera-explorer tests

a61b959

Fix hex_game example

5204ea3

Fix cargo clippy

1ab41d6

deuszx commented Jan 9, 2025

View reviewed changes

linera-chain/src/chain.rs Outdated Show resolved Hide resolved

deuszx commented Jan 9, 2025

View reviewed changes

Cleanup

93afb26

deuszx commented Jan 9, 2025

View reviewed changes

linera-base/src/crypto.rs Outdated Show resolved Hide resolved

deuszx commented Jan 9, 2025

View reviewed changes

linera-core/src/unit_tests/worker_tests.rs Outdated Show resolved Hide resolved

Fix test

bb50374

deuszx force-pushed the new-block-structure branch from 6a9cccf to bb50374 Compare January 9, 2025 18:33

deuszx added 2 commits January 9, 2025 19:35

rm comments

b2c870d

Fix graphql query in the tests

8d06dac

deuszx changed the title ~~New block structure~~ New Block structure Jan 10, 2025

ma2bd reviewed Jan 10, 2025

View reviewed changes

Comments on the new types

d13da68

deuszx force-pushed the new-block-structure branch from dca5da3 to d13da68 Compare January 10, 2025 18:21

deuszx added 2 commits January 10, 2025 19:29

block.rs cleanup

86f300d

Testy empty Block size.

5fe0425

deuszx marked this pull request as ready for review January 10, 2025 18:45

deuszx requested review from afck and bart-linera January 10, 2025 18:47

afck requested changes Jan 11, 2025

View reviewed changes

deuszx added 3 commits January 13, 2025 13:38

Check hash equality when deserializing Block

f71b944

Merge remote-tracking branch 'origin/main' into new-block-structure

ab619da

Address typos and renames

65a80f6

deuszx force-pushed the new-block-structure branch from 200252e to f3c8e49 Compare January 13, 2025 13:22

afck self-requested a review January 13, 2025 13:24

deuszx added 2 commits January 13, 2025 14:42

Fix after merge with main.

19ce2d1

deuszx force-pushed the new-block-structure branch from f3c8e49 to 652604e Compare January 13, 2025 13:42

afck reviewed Jan 13, 2025

View reviewed changes

Update EMPTY_BLOCK_SIZE

8402eb4

deuszx force-pushed the new-block-structure branch from 51161cf to 543802f Compare January 13, 2025 16:35

deuszx requested review from ma2bd and afck January 13, 2025 16:38

Improve comment on the Block type a bit

5cbe42e

deuszx force-pushed the new-block-structure branch from 543802f to 5cbe42e Compare January 13, 2025 16:42

afck reviewed Jan 13, 2025

View reviewed changes

RM (de)serializers for ExecutedBlock

42ae6d2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Block structure #3101

New Block structure #3101

deuszx commented Jan 8, 2025 •

edited

Loading

deuszx Jan 9, 2025

deuszx Jan 9, 2025

deuszx Jan 9, 2025

deuszx Jan 10, 2025

afck Jan 13, 2025

deuszx Jan 13, 2025

ma2bd Jan 10, 2025

deuszx Jan 10, 2025

deuszx Jan 10, 2025

deuszx Jan 10, 2025

ma2bd Jan 10, 2025

deuszx Jan 10, 2025 •

edited

Loading

afck Jan 13, 2025

ma2bd Jan 10, 2025

deuszx Jan 10, 2025 •

edited

Loading

afck Jan 13, 2025

afck Jan 13, 2025

deuszx Jan 13, 2025

afck Jan 13, 2025

afck Jan 13, 2025

afck Jan 13, 2025

afck Jan 13, 2025

afck Jan 13, 2025

deuszx Jan 13, 2025

afck Jan 13, 2025

afck Jan 13, 2025

	/// and operations to executed which define state transition of the chain.
	/// and operations to execute which define a state transition of the chain.

	/// As part of the block body, contains all the incoming messages
	///
	/// As part of the block body, contains all the incoming messages

	.track_block_size(EMPTY_EXECUTED_BLOCK_SIZE)
	.track_block_size(EMPTY_EXECUTED_BLOCK_SIZE)

New Block structure #3101

Are you sure you want to change the base?

New Block structure #3101

Conversation

deuszx commented Jan 8, 2025 • edited Loading

Motivation

Proposal

Test Plan

Release Plan

Links

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deuszx Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deuszx Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deuszx commented Jan 8, 2025 •

edited

Loading

deuszx Jan 10, 2025 •

edited

Loading

deuszx Jan 10, 2025 •

edited

Loading