Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: replace evidence with PFBs in square diagram #1436

Merged
merged 9 commits into from
Mar 3, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion specs/src/rationale/message_block_layout.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,14 @@ Celestia uses [a data availability scheme](https://arxiv.org/abs/1809.09044) tha

## Message Layout Rationale

Block data consists of transactions (which modify the Celestia chain's state), intermediate state roots (required for fraud proofs of the aforementioned transactions), messages (binary blobs which do not modify the Celestia state, but which are intended for a Celestia application identified with a provided namespace ID), and other relevant pieces of data (e.g. evidence for slashing). We want to arrange this data into a `k * k` matrix of fixed-sized shares, which will later be committed to in [Namespace Merkle Trees (NMTs)](../specs/data_structures.md#namespace-merkle-tree).
Block data consists of:

1. Cosmos SDK module transactions (e.g. [MsgSend](https://github.com/cosmos/cosmos-sdk/blob/f71df80e93bffbf7ce5fbd519c6154a2ee9f991b/proto/cosmos/bank/v1beta1/tx.proto#L21-L32)). These modify the Celestia chain's state.
1. Celestia-specific transactions (e.g. [PayForBlobs](../specs/data_structures.md#payforblobdata)). These modify the Celestia chain's state.
1. Intermediate state roots: required for fraud proofs of the aforementioned transactions.
1. Messages: binary blobs which do not modify the Celestia state, but which are intended for a Celestia application identified with a provided namespace ID.

We want to arrange this data into a `k * k` matrix of fixed-sized shares, which will later be committed to in [Namespace Merkle Trees (NMTs)](../specs/data_structures.md#namespace-merkle-tree).

The simplest way we can imagine arranging block data is to simply serialize it all in no particular order, split it into fixed-sized shares, then arrange those shares into the `k * k` matrix in row-major order. However, this naive scheme can be improved in a number of ways, described below.

Expand Down
13 changes: 3 additions & 10 deletions specs/src/specs/consensus.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
- [`block.lastCommit`](#blocklastcommit)
- [`block.availableData`](#blockavailabledata)
- [State Transitions](#state-transitions)
- [`block.availableData.evidenceData`](#blockavailabledataevidencedata)
- [`block.availableData.transactionData`](#blockavailabledatatransactiondata)
- [SignedTransactionDataTransfer](#signedtransactiondatatransfer)
- [SignedTransactionDataMsgPayForData](#signedtransactiondatamsgpayfordata)
Expand Down Expand Up @@ -100,7 +99,7 @@ TODO

The Tendermint consensus protocol is fork-free by construction under an honest majority of stake assumption.

If a block has a [valid commit](#blocklastcommit), it is part of the canonical chain. If equivocation [evidence](./data_structures.md#evidence) is detected for more than 1/3 of voting power, the node must halt. See [rationale doc](../rationale/fork_choice_das.md) for more information.
If a block has a [valid commit](#blocklastcommit), it is part of the canonical chain. If equivocation evidence is detected for more than 1/3 of voting power, the node must halt.

## Block Validity

Expand Down Expand Up @@ -165,7 +164,7 @@ The block's [available data](./data_structures.md#availabledata) (analogous to t
Once parsed, the following checks must be `true`:

1. The commitments of the [erasure-coded extended](./data_structures.md#2d-reed-solomon-encoding-scheme) `availableData` must match those in `header.availableDataHeader`. Implicitly, this means that both rows and columns must be ordered lexicographically by namespace ID since they are committed to in a [Namespace Merkle Tree](data_structures.md#namespace-merkle-tree).
1. Length of `availableData.intermediateStateRootData` == length of `availableData.transactionData` + length of `availableData.evidenceData` + 2. (Two additional state transitions are the [begin](#begin-block) and [end block](#end-block) implicit transitions.)
1. Length of `availableData.intermediateStateRootData` == length of `availableData.transactionData` + length of `availableData.payForBlobData` + 2. (Two additional state transitions are the [begin](#begin-block) and [end block](#end-block) implicit transitions.)
staheri14 marked this conversation as resolved.
Show resolved Hide resolved

## State Transitions

Expand All @@ -176,18 +175,12 @@ For this section, the variable `state` represents the [state tree](./data_struct
State transitions are applied in the following order:

1. [Begin block](#begin-block).
1. [Evidence](#blockavailabledataevidencedata).
1. [Transactions](#blockavailabledatatransactiondata).
1. [End block](#end-block).

### `block.availableData.evidenceData`

Evidence is the second set of state transitions that are applied, ahead of [transactions](#blockavailabledatatransactiondata).
Each evidence represents proof of validator misbehavior, and causes a penalty against the validator(s).

### `block.availableData.transactionData`

Once [evidence has been processed](#blockavailabledataevidencedata), transactions are applied to the state. Note that _transactions_ mutate the state (essentially, the validator set and minimal balances), while _messages_ do not. See [the architecture documentation](./architecture.md) for more info.
Transactions are applied to the state. Note that _transactions_ mutate the state (essentially, the validator set and minimal balances), while _messages_ do not. See [the architecture documentation](./architecture.md) for more info.

`block.availableData.transactionData` is simply a list of [WrappedTransaction](./data_structures.md#wrappedtransaction)s. For each wrapped transaction in this list, `wrappedTransaction`, with index `i` (starting from `0`), the following checks must be `true`:

Expand Down
64 changes: 11 additions & 53 deletions specs/src/specs/data_structures.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,13 +51,10 @@
- [SignedTransactionDataBurn](#signedtransactiondataburn)
- [SignedTransactionRedelegateCommission](#signedtransactionredelegatecommission)
- [SignedTransactionRedelegateReward](#signedtransactionredelegatereward)
- [PayForBlobData](#payforblobdata)
- [IntermediateStateRootData](#intermediatestaterootdata)
- [WrappedIntermediateStateRoot](#wrappedintermediatestateroot)
- [IntermediateStateRoot](#intermediatestateroot)
- [EvidenceData](#evidencedata)
- [Evidence](#evidence)
- [PublicKey](#publickey)
- [Vote](#vote)
- [MessageData](#messagedata)
- [Message](#message)
- [State](#state)
Expand Down Expand Up @@ -152,7 +149,7 @@ Data that is [erasure-coded](#erasure-coding) for [data availability checks](htt
|-----------------------------|---------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|
| `transactionData` | [TransactionData](#transactiondata) | Transaction data. Transactions modify the validator set and balances, and pay fees for messages to be included. |
| `intermediateStateRootData` | [IntermediateStateRootData](#intermediatestaterootdata) | Intermediate state roots used for fraud proofs. |
| `evidenceData` | [EvidenceData](#evidencedata) | Evidence used for slashing conditions (e.g. equivocation). |
| `payForBlobData` | [PayForBlobData](#payforblobdata) | PayForBlob data. Transactions that pay for blobs to be included. |
| `messageData` | [MessageData](#messagedata) | Message data. Messages are app data. |

### Commit
Expand Down Expand Up @@ -188,7 +185,7 @@ Abstraction over transaction fees.

Address is a [type alias](#type-aliases).

Addresses are the [hash](#hashing) [digest](#hashdigest) of the [public key](#publickey).
Addresses are the [hash](#hashing) [digest](#hashdigest) of the [public key](https://docs.cosmos.network/v0.46/basics/accounts.html#public-keys).

Addresses have a length of 32 bytes.

Expand Down Expand Up @@ -246,7 +243,7 @@ Unless otherwise indicated explicitly, objects are first [serialized](#serializa

Consensus-critical data is authenticated using [ECDSA](https://www.secg.org/sec1-v2.pdf), with the curve [secp256k1](https://en.bitcoin.it/wiki/Secp256k1). A highly-optimized library is available in C (<https://github.com/bitcoin-core/secp256k1>), with wrappers in Go (<https://pkg.go.dev/github.com/ethereum/go-ethereum/crypto/secp256k1>) and Rust (<https://docs.rs/crate/secp256k1>).

[Public keys](#publickey) are encoded in uncompressed form, as the concatenation of the `x` and `y` values. No prefix is needed to distinguish between encoding schemes as this is the only encoding supported.
[Public keys](https://docs.cosmos.network/v0.46/basics/accounts.html#public-keys) are encoded in uncompressed form, as the concatenation of the `x` and `y` values. No prefix is needed to distinguish between encoding schemes as this is the only encoding supported.

Deterministic signatures ([RFC-6979](https://tools.ietf.org/rfc/rfc6979.txt)) should be used when signing, but this is not enforced at the protocol level as it cannot be.

Expand Down Expand Up @@ -504,8 +501,8 @@ For shares **with a reserved namespace ID through [`NAMESPACE_ID_MAX_RESERVED`](
- If this is the first share of a sequence, the next [`SEQUENCE_BYTES`](./consensus.md#constants) contain a big endian `uint32` that represents the length of the sequence that follows in bytes.
- The next [`SHARE_RESERVED_BYTES`](./consensus.md#constants) bytes are the starting byte of the length of the [canonically serialized](#serialization) first request that starts in the share, or `0` if there is none, as an unsigned [varint](https://developers.google.com/protocol-buffers/docs/encoding).
- The remaining [`SHARE_SIZE`](./consensus.md#constants)`-`[`NAMESPACE_ID_BYTES`](./consensus.md#constants)`-`[`SHARE_INFO_BYTES`](./consensus.md#constants) `-` [`SEQUENCE_BYTES`](.consensus.md#constants) bytes (only if this is the first share of a sequence) `-` [`SHARE_RESERVED_BYTES`](./consensus.md#constants) bytes are transactions, intermediate state roots, or evidence data depending on the namespace of ths share. Each transaction, intermediate state root, or evidence is prefixed with a [varint](https://developers.google.com/protocol-buffers/docs/encoding) of the length of that unit.
- If there is insufficient transaction, intermediate state root, or evidence data to fill the share, the remaining bytes are filled with `0`.
- The remaining [`SHARE_SIZE`](./consensus.md#constants)`-`[`NAMESPACE_ID_BYTES`](./consensus.md#constants)`-`[`SHARE_INFO_BYTES`](./consensus.md#constants) `-` [`SEQUENCE_BYTES`](.consensus.md#constants) bytes (only if this is the first share of a sequence) `-` [`SHARE_RESERVED_BYTES`](./consensus.md#constants) bytes are transactions, intermediate state roots, or PayForBlob transaction data. Each transaction, intermediate state root, or PayForBlob transaction is prefixed with a [varint](https://developers.google.com/protocol-buffers/docs/encoding) of the length of that unit.
- If there is insufficient transaction, intermediate state root, or PayForBlob transaction data to fill the share, the remaining bytes are filled with `0`.

First share in a sequence:

Expand Down Expand Up @@ -567,17 +564,17 @@ The first [`NAMESPACE_ID_BYTES`](./consensus.md#constants) of a share's raw data

### Arranging Available Data Into Shares

The previous sections described how some original data, arranged into a `k * k` matrix, can be extended into a `2k * 2k` matrix and committed to with NMT roots. This section specifies how [available data](#available-data) (which includes [transactions](#transactiondata), [intermediate state roots](#intermediatestaterootdata), [evidence](#evidencedata), and [messages](#messagedata)) is arranged into the matrix in the first place.
The previous sections described how some original data, arranged into a `k * k` matrix, can be extended into a `2k * 2k` matrix and committed to with NMT roots. This section specifies how [available data](#available-data) (which includes [transactions](#transactiondata), [intermediate state roots](#intermediatestaterootdata), PayForBlob transactions, and [messages](#messagedata)) is arranged into the matrix in the first place.

Then,

1. For each of `transactionData`, `intermediateStateRootData`, and `evidenceData`, [serialize](#serialization):
1. For each of `transactionData`, `intermediateStateRootData`, PayForBlob transactions, [serialize](#serialization):
1. For each request in the list:
1. [Serialize](#serialization) the request (individually).
1. Compute the length of each serialized request, [serialize the length](#serialization), and pre-pend the serialized request with its serialized length.
1. Split up the length/request pairs into [`SHARE_SIZE`](./consensus.md#constants)`-`[`NAMESPACE_ID_BYTES`](./consensus.md#constants)`-`[`SHARE_RESERVED_BYTES`](./consensus.md#constants)-byte chunks.
1. Create a [share](#share) out of each chunk. This data has a _reserved_ namespace ID, so the first [`NAMESPACE_ID_BYTES`](./consensus.md#constants)`+`[`SHARE_RESERVED_BYTES`](./consensus.md#constants) bytes for these shares must be [set specially](#share).
1. Concatenate the lists of shares in the order: transactions, intermediate state roots, evidence.
1. Concatenate the lists of shares in the order: transactions, intermediate state roots, PayForBlob transactions.
rootulp marked this conversation as resolved.
Show resolved Hide resolved

Note that by construction, each share only has a single namespace, and that the list of concatenated shares is [lexicographically ordered by namespace ID](consensus.md#reserved-namespace-ids).

Expand Down Expand Up @@ -797,6 +794,8 @@ Assigns validator's pending commission to a delegation.

Adds delegation's pending rewards to voting power.

### PayForBlobData

### IntermediateStateRootData

| name | type | description |
Expand All @@ -816,47 +815,6 @@ Adds delegation's pending rewards to voting power.
|--------|---------------------------|------------------------------------------------------------------------------------------|
| `root` | [HashDigest](#hashdigest) | Root of intermediate state, which is composed of the global state and the validator set. |

### EvidenceData
staheri14 marked this conversation as resolved.
Show resolved Hide resolved

Wrapper for evidence data.

| name | type | description |
|-------------|---------------------------|------------------------------------------------|
| `evidences` | [Evidence](#evidence)`[]` | List of evidence used for slashing conditions. |

#### Evidence

| name | type | description |
|----------|-------------------------|-------------|
| `pubKey` | [PublicKey](#publickey) | |
| `voteA` | [Vote](#vote) | |
| `voteB` | [Vote](#vote) | |

#### PublicKey

| name | type | description |
|------|------------|--------------------------|
| `x` | `byte[32]` | `x` value of public key. |
| `y` | `byte[32]` | `y` value of public key. |

#### Vote

```C++
enum VoteType : uint8_t {
Prevote = 1,
Precommit = 2,
};
```

| name | type | description |
|--------------|---------------------------|-------------|
| `type` | `VoteType` | |
| `height` | [Height](#type-aliases) | |
| `round` | [Round](#type-aliases) | |
| `headerHash` | [HashDigest](#hashdigest) | |
| `timestamp` | [Timestamp](#timestamp) | |
| `signature` | [Signature](#signature) | |

### MessageData

| name | type | description |
Expand Down
4 changes: 2 additions & 2 deletions specs/src/specs/figures/block_data_structures.dot
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ digraph G {

subgraph cluster_body {
label = "availableData";
struct3 [label = "{ | { transactionData | intermediateStateRoots | evidenceData | messageData } }"];
struct3 [label = "{ | { transactionData | intermediateStateRoots | payForBlobData | messageData } }"];
}

subgraph cluster_lastcommit {
Expand All @@ -33,4 +33,4 @@ digraph G {
edge [style = invis];
struct1 -> struct3;
struct1 -> struct4;
}
}
20 changes: 7 additions & 13 deletions specs/src/specs/figures/block_data_structures.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion specs/src/specs/figures/rs2d_originaldata_message.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading