Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: add block validity rules specs #1966

Merged
merged 24 commits into from
Jul 6, 2023
Merged

docs: add block validity rules specs #1966

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
134b2b4
docs: init block validaity rules
evan-forbes Jun 7, 2023
0555466
docs: flesh out some of the rules
evan-forbes Jun 11, 2023
f3a5470
Merge branch 'main' into evan/block-validity-rules
evan-forbes Jun 20, 2023
51411fa
docs: add general summary of block validity rules
evan-forbes Jun 20, 2023
07c2678
chore: revert linter change
evan-forbes Jun 20, 2023
8173c9c
chore: clean up
evan-forbes Jun 20, 2023
b449608
fix: typo
evan-forbes Jun 20, 2023
d0329fd
fix: add feedback
evan-forbes Jun 23, 2023
069bbd7
fix: clarify multiple blobs per blobtx
evan-forbes Jun 23, 2023
8a5a7d1
chore: add link to data square layout
evan-forbes Jun 23, 2023
f8660a7
Merge branch 'main' into evan/block-validity-rules
evan-forbes Jun 23, 2023
b6ebd26
Merge branch 'main' into evan/block-validity-rules
evan-forbes Jun 23, 2023
4cac7a4
fix: typo
evan-forbes Jun 24, 2023
c3e19fe
refactor: move fraud proofs to their own section
evan-forbes Jun 26, 2023
57380e7
chore: add fraud proofs section to readme and summary
evan-forbes Jun 26, 2023
f72be81
fix: typo
evan-forbes Jun 26, 2023
550e4f5
Merge branch 'main' into evan/block-validity-rules
evan-forbes Jun 26, 2023
b36dcc4
Merge branch 'main' into evan/block-validity-rules
evan-forbes Jun 26, 2023
89c03ac
docs: clarifications
evan-forbes Jun 29, 2023
88a3ac7
fix: better wording around light clients
evan-forbes Jul 5, 2023
27209c8
docs: remove duplicate specs and just link to things in the validity …
evan-forbes Jul 5, 2023
9dca098
docs: simplyfy further
evan-forbes Jul 5, 2023
71cac1e
docs: minor refactor
evan-forbes Jul 5, 2023
7ac6ec3
chore: typo
evan-forbes Jul 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions specs/src/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
- [Consensus](./specs/consensus.md)
- [Block Proposer](./specs/block_proposer.md)
- [Block Validity Rules](./specs/block_validity_rules.md)
- [Fraud Proofs](./specs/fraud_proofs.md)
- [Networking](./specs/networking.md)
- [Public-Key Cryptography](./specs/public_key_cryptography.md)
- [Data Square Layout](./specs/data_square_layout.md)
Expand Down
1 change: 1 addition & 0 deletions specs/src/SUMMARY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
- [Consensus](./specs/consensus.md)
- [Block Proposer](./specs/block_proposer.md)
- [Block Validity Rules](./specs/block_validity_rules.md)
- [Fraud Proofs](./specs/fraud_proofs.md)
- [Networking](./specs/networking.md)
- [Public-Key Cryptography](./specs/public_key_cryptography.md)
- [Data Square Layout](./specs/data_square_layout.md)
Expand Down
38 changes: 8 additions & 30 deletions specs/src/specs/block_validity_rules.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@ Unlike most blockchains, Celestia derives most of its functionality from
stateless commitments to data rather than stateful transitions. This means that
the protocol relies heavily on block validity rules. Notably, resource
constrained light clients must be able to detect when these validity rules have
not been followed in order to avoid making an honest majority assumption.
not been followed in order to avoid making an honest majority assumption. This
evan-forbes marked this conversation as resolved.
Show resolved Hide resolved
has a significant impact on thier design. More information on how light clients verify
block validity rules can be foud in the [Fraud Proofs](./fraud_proofs.md) spec.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Question] Do light nodes actually verify all the block validity rules? or just a subset? To me, it is more like the light client can only check the "Invalidity of the blocks" but not the validity. And they even do so with the aid of full nodes.


> **Note** Celestia relies on CometBFT (formerly tendermint) for consensus,
> meaning that it has single slot finality and is fork-free. Therefore, in order
Expand All @@ -14,7 +16,7 @@ not been followed in order to avoid making an honest majority assumption.
> proofs are created for light clients. After light clients verify fraud proofs,
> they halt.

Before any Celestia specific validation is performed, all cometBFT [block
Before any Celestia specific validation is performed, all CometBFT [block
validation
rules](https://github.com/cometbft/cometbft/blob/v0.34.28/spec/core/data_structures.md#block)
must be followed. The only deviation from these rules is how the data root
Expand All @@ -26,7 +28,7 @@ change, including how it proves data availability to light clients.

The data for each block must be considered available before a given block can be
considered valid. For consensus nodes, this is done via an identical mechanism
to a normal cometBFT node, which involves downloading the entire block by each
to a normal CometBFT node, which involves downloading the entire block by each
node before considering that block valid.

Light clients however do not download the entire block. They only sample a
Expand All @@ -39,27 +41,15 @@ Majorities"](https://arxiv.org/abs/1809.09044) and in the
Per the [LazyLedger white paper](https://arxiv.org/pdf/1905.09274.pdf), Celestia
uses a 2D Reed-Solomon coding scheme
([rsmt2d](https://github.com/celestiaorg/rsmt2d)) to accommodate data
availability sampling. This involves "splitting" the cometBFT block data into
availability sampling. This involves "splitting" the CometBFT block data into
shares. Along with the 2D scheme, Celestia also makes use of [namespaced merkle
trees (nmt)](https://github.com/celestiaorg/nmt). These are combined to create
the commitment over block data instead of the typical merkle tree used by
cometBFT.
CometBFT.

<img src="./figures/data_root.svg" alt="Figure 1: Data Root" width="400"/> <img
src="./figures/rs2d_quadrants.svg" alt="Figure 2: rsmt2d" width="400"/>

### Bad Encoding Fraud Proofs

In order for data availability sampling to work, light clients must be convinced
that erasure encoded parity data was encoded correctly. For light clients, this
is ultimately enforced via [bad encoding fraud proofs
(BEFPs)](https://github.com/celestiaorg/celestia-node/blob/v0.11.0-rc3/docs/adr/adr-006-fraud-service.md#detailed-design).
Consensus nodes must verify this themselves before considering a block valid.
This is done automatically by verifying the data root of the header, since that
requires reconstructing the square from the block data, performing the erasure
encoding, calculating the data root using that representation, and then
comparing the data root found in the header.

### Square Construction

The construction of the square is critical in providing additional guarantees to
Expand All @@ -75,7 +65,7 @@ Each chunk of block data is split into equally size shares for sampling
purposes. The encoding was designed to allow for light clients to decode these
shares to retrieve relevant data and to be future-proof yet backwards
compatible. The share encoding is deeply integrated into square construction, and
therefore critical to calculate the data root.
therefore critical to calculate the data root.

See [shares spec](./shares.md)

Expand All @@ -85,15 +75,3 @@ Each `BlobTx` consists of a transaction to pay for one or more blobs, and the
blobs themselves. Each `BlobTx` that is included in the block must be valid.
Those rules are described in [`x/blob` module
specs](../../../x/blob/README.md#validity-rules)

## Blob Inclusion

TODO

## State Fraud Proofs

State fraud proofs allow light clients to avoid making an honest majority for
state validity. While these are not incorporated into the protocol as of v1.0.0,
there are example implementations that can be found in
[Rollkit](https://github.com/rollkit/rollkit). More info in
[rollkit-ADR009](https://github.com/rollkit/rollkit/blob/4fd97ba8b8352771f2e66454099785d06fd0c31b/docs/lazy-adr/adr-009-state-fraud-proofs.md).
25 changes: 25 additions & 0 deletions specs/src/specs/fraud_proofs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Fraud Proofs

## Bad Encoding Fraud Proofs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Question] I think the story of light clients and how they consider a block valid actually deserves separate specs, and I guess it is not even part of the core/app implementation, right? which means it can live in the node repo. I think here it's better to focus on the entities that actually exist in the core/app layer e.g., full nodes, validators, consensus nodes, etc. While the content of this part adds additional great info I think is not directly relevant. Due to this, and in favor of brevity and conciseness, I'd suggest leaving this subsection outside of the current specs. Though, I'll leave it up to you.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a good point that fraud proofs are not directly part of core/app atm, so including them in the spec might not make a lot of sense. I do think we should at least have a link to the relevant specs, since that will preserve our capability to send a newcomer a single link where they can find all the important portions.

The initial reason for including something here is that I feel that they have a large impact on the design and implementation. For example, the sole reason we have square layout/blob commitment rules are to accomodate blob inclusion proofs. If we eventually have at least a high level description of it, then we can see the end result, which I think makes the rest of the concepts a lot clearer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think we should at least have a link to the relevant specs, since that will preserve our capability to send a newcomer a single link where they can find all the important portions.

Good idea and agree!

The initial reason for including something here is that I feel that they have a large impact on the design and implementation. For example, the sole reason we have square layout/blob commitment rules are to accomodate blob inclusion proofs. If we eventually have at least a high level description of it, then we can see the end result, which I think makes the rest of the concepts a lot clearer.

Agree that such information, at a high level, will provide readers with a clear understanding of the motivations behind the design choices and the purpose of the validity rules.
Following my previous comment, I recommend incorporating all the motivational information into a single section. Additionally, it would be beneficial to explicitly mention that the design of the Celestia block is influenced by these reasons, resulting in an extensive list of validity rules.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think that wherever validity rules are described should be were fraud proofs which define the failure to comply with the aforementioned validity rules should reside. Therefore I think fraud proofs should be described in the specification here.


In order for data availability sampling to work, light clients must be convinced
that erasure encoded parity data was encoded correctly. For light clients, this
is ultimately enforced via [bad encoding fraud proofs
(BEFPs)](https://github.com/celestiaorg/celestia-node/blob/v0.11.0-rc3/docs/adr/adr-006-fraud-service.md#detailed-design).
Consensus nodes must verify this themselves before considering a block valid.
This is done automatically by verifying the data root of the header, since that
requires reconstructing the square from the block data, performing the erasure
encoding, calculating the data root using that representation, and then
comparing the data root found in the header.

## Blob Inclusion

TODO

## State

State fraud proofs allow light clients to avoid making an honest majority for
state validity. While these are not incorporated into the protocol as of v1.0.0,
evan-forbes marked this conversation as resolved.
Show resolved Hide resolved
there are example implementations that can be found in
[Rollkit](https://github.com/rollkit/rollkit). More info in
[rollkit-ADR009](https://github.com/rollkit/rollkit/blob/4fd97ba8b8352771f2e66454099785d06fd0c31b/docs/lazy-adr/adr-009-state-fraud-proofs.md).