From e445aaca88ebdd198160ef188673f5079ddd42cf Mon Sep 17 00:00:00 2001 From: sanaz Date: Mon, 17 Jul 2023 15:44:04 -0700 Subject: [PATCH 01/20] expands on the reserved namespaces section --- specs/src/specs/namespace.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index b259516cdb..9e3cdc6127 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -16,9 +16,10 @@ A namespace is composed of two fields: [version](#version) and [id](#id). A name ### Version -The namespace version is an 8-bit unsigned integer that indicates the version of the namespace. The version is used to determine the format of the namespace. The only supported user-specifiable namespace version is `0`. The version is encoded as a single byte. - -Note: The `PARITY_SHARE_NAMESPACE` uses the namespace version `255` so that it can be ignored via the `IgnoreMaxNamespace` feature from [nmt](https://github.com/celestiaorg/nmt). The `TAIL_PADDING_NAMESPACE` uses the namespace version `255` so that it remains ordered after all blob namespaces even in the case a new namespace version is introduced. +The namespace version is an 8-bit unsigned integer that indicates the version of the namespace. +The version is used to determine the format of the namespace. +The only supported user-specifiable namespace version is `0`. +The version is encoded as a single byte. A namespace with version `0` must contain an id with a prefix of 18 leading `0` bytes. The remaining 10 bytes of the id are user-specified. @@ -41,6 +42,14 @@ A new namespace version MUST be introduced if the namespace format changes in a The namespace ID is a 28 byte identifier that uniquely identifies a namespace. The ID is encoded as a byte slice of length 28. ## Reserved Namespaces +Celestia reserves certain namespaces with specific meanings. +As a result, applications must refrain from using these reserved namespaces for their data. +The rationale behind this is that these namespaces dictate the positioning of data within the Celestia block. +Thus, it is advisable for applications to avoid utilizing these reserved namespaces to ensure the desired placement of their data. +Below is the list of reserved namespaces with a description of their meaning. +For more details regarding the meaning and application of the reserved namespaces, please see the [Data Square Layout](./data-square-layout.md) section. +[//]: # (Note: The `PARITY_SHARE_NAMESPACE` uses the namespace version `255` so that it can be ignored via the `IgnoreMaxNamespace` feature from [nmt](https://github.com/celestiaorg/nmt). The `TAIL_PADDING_NAMESPACE` uses the namespace version `255` so that it remains ordered after all blob namespaces even in the case a new namespace version is introduced.) + | name | type | value | description | |-------------------------------------|-------------|----------------------------------------------------------------|------------------------------------------------------------------------------------------------------| From 8de047aa53558c42166d10de1830ebcfd7da7338 Mon Sep 17 00:00:00 2001 From: sanaz Date: Mon, 17 Jul 2023 16:01:29 -0700 Subject: [PATCH 02/20] moves version 0 under its own subsection --- specs/src/specs/namespace.md | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index 9e3cdc6127..8785a4d255 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -4,28 +4,32 @@ ## Abstract -One of Celestia's core data structures is the namespace. When a user submits a `MsgPayForBlobs` transaction to Celestia they MUST associate each blob with exactly one namespace. After their transaction has been included in a block, the namespace enables users to take an interest in a subset of the blobs published to Celestia by allowing the user to query for blobs by namespace. +One of Celestia's core data structures is the namespace. When a user submits a transaction encapsulating a `MsgPayForBlobs` message to Celestia, they MUST associate each blob with exactly one namespace. After their transaction has been included in a block, the namespace enables users to take an interest in a subset of the blobs published to Celestia by allowing the user to query for blobs by namespace. In order to enable efficient retrieval of blobs by namespace, Celestia makes use of a [Namespaced Merkle Tree](https://github.com/celestiaorg/nmt). See section 5.2 of the [LazyLedger whitepaper](https://arxiv.org/pdf/1905.09274.pdf) for more details. ## Overview -A namespace is composed of two fields: [version](#version) and [id](#id). A namespace is encoded as a byte slice with the version and id concatenated. Each [share](./shares.md) is prefixed with exactly one namespace. +A namespace is composed of two fields: [version](#version) and [id](#id). A namespace is encoded as a byte slice with the version and id concatenated. ![namespace](./figures/namespace.svg) ### Version The namespace version is an 8-bit unsigned integer that indicates the version of the namespace. -The version is used to determine the format of the namespace. -The only supported user-specifiable namespace version is `0`. +The version is used to determine the format of the namespace. The version is encoded as a single byte. -A namespace with version `0` must contain an id with a prefix of 18 leading `0` bytes. The remaining 10 bytes of the id are user-specified. +#### Version 0 +The only supported user-specifiable namespace version is `0`. +A namespace with version `0` MUST contain an id with a prefix of 18 leading `0` bytes. The remaining 10 bytes of the id are user-specified. + +While version `0` is the only supported user-specifiable namespace version, Celestia MAY utilize other namespace versions for internal use. +For more details, see the [Reserved Namespaces](#reserved-namespaces) section. ```go // Valid encoded namespaces -0x0000000000000000000000000000000000000000000000000000000001 // transaction namespace +0x0000000000000000000000000000000000000000000000000000000001 // transaction namespace [?] is this user-specified? I mean, when sending a transaction, does a user need to associate this namespace before submission of the tx to a validator? or is this namespace added later during the data square construction? 0x0000000000000000000000000000000000000001010101010101010101 // valid blob namespace 0x0000000000000000000000000000000000000011111111111111111111 // valid blob namespace @@ -43,10 +47,10 @@ The namespace ID is a 28 byte identifier that uniquely identifies a namespace. T ## Reserved Namespaces Celestia reserves certain namespaces with specific meanings. -As a result, applications must refrain from using these reserved namespaces for their data. +As a result, applications MUST refrain from using these reserved namespaces for their blob data. The rationale behind this is that these namespaces dictate the positioning of data within the Celestia block. -Thus, it is advisable for applications to avoid utilizing these reserved namespaces to ensure the desired placement of their data. -Below is the list of reserved namespaces with a description of their meaning. +Thus, applications ensure the desired placement of their blob data by avoiding utilizing these reserved namespaces. +Below is the list of reserved namespaces with a brief description of their meaning. For more details regarding the meaning and application of the reserved namespaces, please see the [Data Square Layout](./data-square-layout.md) section. [//]: # (Note: The `PARITY_SHARE_NAMESPACE` uses the namespace version `255` so that it can be ignored via the `IgnoreMaxNamespace` feature from [nmt](https://github.com/celestiaorg/nmt). The `TAIL_PADDING_NAMESPACE` uses the namespace version `255` so that it remains ordered after all blob namespaces even in the case a new namespace version is introduced.) From 29888be0254d8973ddf2e0f2e3990ff946e91e37 Mon Sep 17 00:00:00 2001 From: sanaz Date: Mon, 17 Jul 2023 16:07:37 -0700 Subject: [PATCH 03/20] reorganizes the version subsection and expands on the version 0 --- specs/src/specs/namespace.md | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index 8785a4d255..cbe49df96c 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -16,17 +16,22 @@ A namespace is composed of two fields: [version](#version) and [id](#id). A name ### Version -The namespace version is an 8-bit unsigned integer that indicates the version of the namespace. -The version is used to determine the format of the namespace. +The namespace version is an 8-bit unsigned integer that indicates the version of the namespace. The version is encoded as a single byte. +The version is used to determine the format of the namespace. +A new namespace version MUST be introduced if the namespace format changes in a backwards incompatible way. + + +Below we enumerate and explain supported user-specifiable namespace version, however, we note that Celestia MAY utilize other namespace versions for internal use. +For more details, see the [Reserved Namespaces](#reserved-namespaces) section. #### Version 0 The only supported user-specifiable namespace version is `0`. -A namespace with version `0` MUST contain an id with a prefix of 18 leading `0` bytes. The remaining 10 bytes of the id are user-specified. +A namespace with version `0` MUST contain an id with a prefix of 18 leading `0` bytes. +The remaining 10 bytes of the id are user-specified. +Below, we provide examples of valid and invalid encoded user-supplied namespaces with version `0`. -While version `0` is the only supported user-specifiable namespace version, Celestia MAY utilize other namespace versions for internal use. -For more details, see the [Reserved Namespaces](#reserved-namespaces) section. ```go // Valid encoded namespaces 0x0000000000000000000000000000000000000000000000000000000001 // transaction namespace [?] is this user-specified? I mean, when sending a transaction, does a user need to associate this namespace before submission of the tx to a validator? or is this namespace added later during the data square construction? @@ -38,12 +43,12 @@ For more details, see the [Reserved Namespaces](#reserved-namespaces) section. 0x1000000000000000000000000000000000000000000000000000000000 // invalid because it does not have version 0 0x1111111111111111111111111111111111111111111111111111111111 // invalid because it does not have version 0 ``` - -A new namespace version MUST be introduced if the namespace format changes in a backwards incompatible way (i.e. the number of leading `0` bytes in the id prefix is reduced). +Any change in the number of leading `0` bytes in the id of a namespace with version `0` is considered a backwards incompatible change and MUST be introduced as a new namespace version. ### ID -The namespace ID is a 28 byte identifier that uniquely identifies a namespace. The ID is encoded as a byte slice of length 28. +The namespace ID is a 28 byte identifier that uniquely identifies a namespace. +The ID is encoded as a byte slice of length 28. ## Reserved Namespaces Celestia reserves certain namespaces with specific meanings. @@ -71,6 +76,10 @@ For more details regarding the meaning and application of the reserved namespace See [pkg/namespace](../../../pkg/namespace). + +## Protobuf Definition + +[//]: # (TODO: Add protobuf definition for namespace) ## References 1. [ADR-014](../../../docs/architecture/adr-014-versioned-namespaces.md) From 421c4411d357843e3f57fa6c0da783b9765f7797 Mon Sep 17 00:00:00 2001 From: sanaz Date: Mon, 17 Jul 2023 16:14:00 -0700 Subject: [PATCH 04/20] explains the usage of the version value 255 --- specs/src/specs/namespace.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index cbe49df96c..c48cab843d 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -52,13 +52,14 @@ The ID is encoded as a byte slice of length 28. ## Reserved Namespaces Celestia reserves certain namespaces with specific meanings. -As a result, applications MUST refrain from using these reserved namespaces for their blob data. +Applications MUST refrain from using these reserved namespaces for their blob data. The rationale behind this is that these namespaces dictate the positioning of data within the Celestia block. Thus, applications ensure the desired placement of their blob data by avoiding utilizing these reserved namespaces. -Below is the list of reserved namespaces with a brief description of their meaning. +Below is the list of reserved namespaces with a brief description of each. +As you can see in the table, the `PARITY_SHARE_NAMESPACE` and the `TAIL_PADDING_NAMESPACE` use the namespace version `255`, which is different from the supported user-specified versions. +This particular version can be ignored via the `IgnoreMaxNamespace` feature from [nmt](https://github.com/celestiaorg/nmt). +Likewise, the `TAIL_PADDING_NAMESPACE` uses the namespace version `255` so that it remains ordered after all blob namespaces even in the case a new namespace version is introduced. For more details regarding the meaning and application of the reserved namespaces, please see the [Data Square Layout](./data-square-layout.md) section. -[//]: # (Note: The `PARITY_SHARE_NAMESPACE` uses the namespace version `255` so that it can be ignored via the `IgnoreMaxNamespace` feature from [nmt](https://github.com/celestiaorg/nmt). The `TAIL_PADDING_NAMESPACE` uses the namespace version `255` so that it remains ordered after all blob namespaces even in the case a new namespace version is introduced.) - | name | type | value | description | |-------------------------------------|-------------|----------------------------------------------------------------|------------------------------------------------------------------------------------------------------| From 8549b305ed46b4e61c5440d7b6458002e64da0a4 Mon Sep 17 00:00:00 2001 From: sanaz Date: Tue, 18 Jul 2023 10:49:19 -0700 Subject: [PATCH 05/20] conforms to semantic line breaks --- specs/src/specs/namespace.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index c48cab843d..809d3a7a02 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -4,13 +4,17 @@ ## Abstract -One of Celestia's core data structures is the namespace. When a user submits a transaction encapsulating a `MsgPayForBlobs` message to Celestia, they MUST associate each blob with exactly one namespace. After their transaction has been included in a block, the namespace enables users to take an interest in a subset of the blobs published to Celestia by allowing the user to query for blobs by namespace. +One of Celestia's core data structures is the namespace. +When a user submits a transaction encapsulating a `MsgPayForBlobs` message to Celestia, they MUST associate each blob with exactly one namespace. +After their transaction has been included in a block, the namespace enables users to take an interest in a subset of the blobs published to Celestia by allowing the user to query for blobs by namespace. -In order to enable efficient retrieval of blobs by namespace, Celestia makes use of a [Namespaced Merkle Tree](https://github.com/celestiaorg/nmt). See section 5.2 of the [LazyLedger whitepaper](https://arxiv.org/pdf/1905.09274.pdf) for more details. +In order to enable efficient retrieval of blobs by namespace, Celestia makes use of a [Namespaced Merkle Tree](https://github.com/celestiaorg/nmt). +See section 5.2 of the [LazyLedger whitepaper](https://arxiv.org/pdf/1905.09274.pdf) for more details. ## Overview -A namespace is composed of two fields: [version](#version) and [id](#id). A namespace is encoded as a byte slice with the version and id concatenated. +A namespace is composed of two fields: [version](#version) and [id](#id). +A namespace is encoded as a byte slice with the version and id concatenated. ![namespace](./figures/namespace.svg) @@ -80,8 +84,11 @@ See [pkg/namespace](../../../pkg/namespace). ## Protobuf Definition -[//]: # (TODO: Add protobuf definition for namespace) +[//]: # (TODO: Add protobuf definition for namespace if any) ## References 1. [ADR-014](../../../docs/architecture/adr-014-versioned-namespaces.md) 1. [ADR-015](../../../docs/architecture/adr-015-namespace-id-size.md) +1. [Namespaced Merkle Tree](https://github.com/celestiaorg/nmt) +1. [LazyLedger whitepaper](https://arxiv.org/pdf/1905.09274.pdf) +1. [Data Square Layout](./data-square-layout.md) From 39afd725e9fae9d9b19b64240b60a727d749e2a2 Mon Sep 17 00:00:00 2001 From: sanaz Date: Tue, 18 Jul 2023 11:08:47 -0700 Subject: [PATCH 06/20] adds one item under considerations --- specs/src/specs/namespace.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index 809d3a7a02..9bc9e580d0 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -26,12 +26,11 @@ The version is used to determine the format of the namespace. A new namespace version MUST be introduced if the namespace format changes in a backwards incompatible way. -Below we enumerate and explain supported user-specifiable namespace version, however, we note that Celestia MAY utilize other namespace versions for internal use. +Below we explain supported user-specifiable namespace versions, however, we note that Celestia MAY utilize other namespace versions for internal use. For more details, see the [Reserved Namespaces](#reserved-namespaces) section. #### Version 0 The only supported user-specifiable namespace version is `0`. - A namespace with version `0` MUST contain an id with a prefix of 18 leading `0` bytes. The remaining 10 bytes of the id are user-specified. Below, we provide examples of valid and invalid encoded user-supplied namespaces with version `0`. @@ -77,6 +76,8 @@ For more details regarding the meaning and application of the reserved namespace ## Assumptions and Considerations +Application MUST refrain from using the reserved namespaces for their blob data. + ## Implementation See [pkg/namespace](../../../pkg/namespace). From 1f46ff5a340659efdf5b2acab840387717c0735e Mon Sep 17 00:00:00 2001 From: sanaz Date: Tue, 18 Jul 2023 11:19:56 -0700 Subject: [PATCH 07/20] adds a link to the reserved namespaces --- specs/src/specs/namespace.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index 9bc9e580d0..1babc12680 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -76,7 +76,7 @@ For more details regarding the meaning and application of the reserved namespace ## Assumptions and Considerations -Application MUST refrain from using the reserved namespaces for their blob data. +Applications MUST refrain from using the [reserved namespaces](#reserved-namespaces) for their blob data. ## Implementation From 96e2140ffab9ec6b67977c428ddd34f5ce3098b5 Mon Sep 17 00:00:00 2001 From: sanaz Date: Tue, 18 Jul 2023 12:45:10 -0700 Subject: [PATCH 08/20] revises the reserved namespaces section --- specs/src/specs/namespace.md | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index 1babc12680..4f1217d586 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -21,12 +21,14 @@ A namespace is encoded as a byte slice with the version and id concatenated. ### Version The namespace version is an 8-bit unsigned integer that indicates the version of the namespace. -The version is encoded as a single byte. -The version is used to determine the format of the namespace. +The version is used to determine the format of the namespace and +is encoded as a single byte. +[//]: # (TODO: It may be useful to indicate the endianness of the encoding) A new namespace version MUST be introduced if the namespace format changes in a backwards incompatible way. -Below we explain supported user-specifiable namespace versions, however, we note that Celestia MAY utilize other namespace versions for internal use. +Below we explain supported user-specifiable namespace versions, +however, we note that Celestia MAY utilize other namespace versions for internal use. For more details, see the [Reserved Namespaces](#reserved-namespaces) section. #### Version 0 @@ -51,18 +53,21 @@ Any change in the number of leading `0` bytes in the id of a namespace with vers ### ID The namespace ID is a 28 byte identifier that uniquely identifies a namespace. -The ID is encoded as a byte slice of length 28. +The ID is encoded as a byte slice of length 28. + +[//]: # (TODO: It may be useful to indicate the endianness of the encoding) ## Reserved Namespaces Celestia reserves certain namespaces with specific meanings. -Applications MUST refrain from using these reserved namespaces for their blob data. +Applications MUST refrain from using these reserved namespaces for their blob data. [//]: # (TODO: the implication of this is still under investigation, but it is prudent to advise app developers not using this namespace) The rationale behind this is that these namespaces dictate the positioning of data within the Celestia block. -Thus, applications ensure the desired placement of their blob data by avoiding utilizing these reserved namespaces. -Below is the list of reserved namespaces with a brief description of each. -As you can see in the table, the `PARITY_SHARE_NAMESPACE` and the `TAIL_PADDING_NAMESPACE` use the namespace version `255`, which is different from the supported user-specified versions. -This particular version can be ignored via the `IgnoreMaxNamespace` feature from [nmt](https://github.com/celestiaorg/nmt). -Likewise, the `TAIL_PADDING_NAMESPACE` uses the namespace version `255` so that it remains ordered after all blob namespaces even in the case a new namespace version is introduced. -For more details regarding the meaning and application of the reserved namespaces, please see the [Data Square Layout](./data-square-layout.md) section. +Thus, by avoiding utilizing these reserved namespaces, applications ensure the desired placement of their blob data. + +Below is a list of reserved namespaces, along with a brief description of each. +In the table, you will notice that the `PARITY_SHARE_NAMESPACE` and `TAIL_PADDING_NAMESPACE` utilize the namespace version `255`, which differs from the supported user-specified versions. +The reason for employing version `255` for the `PARITY_SHARE_NAMESPACE` is to enable more efficient proof generation within the context of [nmt](https://github.com/celestiaorg/nmt), where it is used in conjunction with the `IgnoreMaxNamespace` feature. +Similarly, the `TAIL_PADDING_NAMESPACE` utilizes the namespace version `255` to ensure that padding shares are always properly ordered and placed at the end of the square even if a new namespace version is introduced. +For additional information on the significance and application of the reserved namespaces, please refer to the [Data Square Layout](./data-square-layout.md) section. | name | type | value | description | |-------------------------------------|-------------|----------------------------------------------------------------|------------------------------------------------------------------------------------------------------| From ac657d198a27be911fc7f90a339e9b39e7aefc87 Mon Sep 17 00:00:00 2001 From: sanaz Date: Tue, 18 Jul 2023 12:49:24 -0700 Subject: [PATCH 09/20] replaces MUST with SHOULD --- specs/src/specs/namespace.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index 4f1217d586..c63154c0b1 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -59,14 +59,14 @@ The ID is encoded as a byte slice of length 28. ## Reserved Namespaces Celestia reserves certain namespaces with specific meanings. -Applications MUST refrain from using these reserved namespaces for their blob data. [//]: # (TODO: the implication of this is still under investigation, but it is prudent to advise app developers not using this namespace) -The rationale behind this is that these namespaces dictate the positioning of data within the Celestia block. +Applications SHOULD refrain from using these reserved namespaces for their blob data. [//]: # (TODO: the implication of this is still under investigation, but it is prudent to advise app developers not using this namespace) +The rationale behind this is that these namespaces dictate the positioning of transactions and blobs data within the Celestia block. Thus, by avoiding utilizing these reserved namespaces, applications ensure the desired placement of their blob data. Below is a list of reserved namespaces, along with a brief description of each. In the table, you will notice that the `PARITY_SHARE_NAMESPACE` and `TAIL_PADDING_NAMESPACE` utilize the namespace version `255`, which differs from the supported user-specified versions. The reason for employing version `255` for the `PARITY_SHARE_NAMESPACE` is to enable more efficient proof generation within the context of [nmt](https://github.com/celestiaorg/nmt), where it is used in conjunction with the `IgnoreMaxNamespace` feature. -Similarly, the `TAIL_PADDING_NAMESPACE` utilizes the namespace version `255` to ensure that padding shares are always properly ordered and placed at the end of the square even if a new namespace version is introduced. +Similarly, the `TAIL_PADDING_NAMESPACE` utilizes the namespace version `255` to ensure that padding shares are always properly ordered and placed at the end of the Celestia data square even if a new namespace version is introduced. For additional information on the significance and application of the reserved namespaces, please refer to the [Data Square Layout](./data-square-layout.md) section. | name | type | value | description | @@ -81,7 +81,7 @@ For additional information on the significance and application of the reserved n ## Assumptions and Considerations -Applications MUST refrain from using the [reserved namespaces](#reserved-namespaces) for their blob data. +Applications SHOULD refrain from using the [reserved namespaces](#reserved-namespaces) for their blob data. ## Implementation From 713a160b2de986dcd743ae51bc0f41b1ff22bfe7 Mon Sep 17 00:00:00 2001 From: sanaz Date: Tue, 18 Jul 2023 12:52:24 -0700 Subject: [PATCH 10/20] replaces block with the proper term of data square --- specs/src/specs/namespace.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index c63154c0b1..b06f698f21 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -60,8 +60,8 @@ The ID is encoded as a byte slice of length 28. ## Reserved Namespaces Celestia reserves certain namespaces with specific meanings. Applications SHOULD refrain from using these reserved namespaces for their blob data. [//]: # (TODO: the implication of this is still under investigation, but it is prudent to advise app developers not using this namespace) -The rationale behind this is that these namespaces dictate the positioning of transactions and blobs data within the Celestia block. -Thus, by avoiding utilizing these reserved namespaces, applications ensure the desired placement of their blob data. +The rationale behind this is that these namespaces dictate the positioning of transactions and blobs data within the Celestia [data square](./data-square-layout.md). +Therefore, by refraining from using these reserved namespaces, applications can guarantee the desired arrangement of their blob data. Below is a list of reserved namespaces, along with a brief description of each. In the table, you will notice that the `PARITY_SHARE_NAMESPACE` and `TAIL_PADDING_NAMESPACE` utilize the namespace version `255`, which differs from the supported user-specified versions. From 27dd594ce562c2a24d2d3aec7efcd3a667997bb2 Mon Sep 17 00:00:00 2001 From: sanaz Date: Tue, 18 Jul 2023 12:53:05 -0700 Subject: [PATCH 11/20] removes excess line --- specs/src/specs/namespace.md | 1 - 1 file changed, 1 deletion(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index b06f698f21..6af6ccdc75 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -54,7 +54,6 @@ Any change in the number of leading `0` bytes in the id of a namespace with vers The namespace ID is a 28 byte identifier that uniquely identifies a namespace. The ID is encoded as a byte slice of length 28. - [//]: # (TODO: It may be useful to indicate the endianness of the encoding) ## Reserved Namespaces From f25dee0d8f18ef106231d115a4a7607ca60da1c9 Mon Sep 17 00:00:00 2001 From: sanaz Date: Tue, 18 Jul 2023 13:13:21 -0700 Subject: [PATCH 12/20] fixes linter issues --- specs/src/specs/data_structures.md | 3 +-- specs/src/specs/fraud_proofs.md | 2 +- specs/src/specs/namespace.md | 34 ++++++++++++++++-------------- tools/blocktime/readme.md | 6 +++--- 4 files changed, 23 insertions(+), 22 deletions(-) diff --git a/specs/src/specs/data_structures.md b/specs/src/specs/data_structures.md index 4f00e2f6db..c07d630cba 100644 --- a/specs/src/specs/data_structures.md +++ b/specs/src/specs/data_structures.md @@ -213,8 +213,7 @@ A proof for a leaf in a [binary Merkle tree](#binary-merkle-tree), as per Sectio ### Namespace Merkle Tree - - + [Shares](./shares.md) in Celestia are associated with a provided _namespace_. The Namespace Merkle Tree (NMT) is a variation of the [Merkle Interval Tree](https://eprint.iacr.org/2018/642), which is itself an extension of the [Merkle Sum Tree](https://bitcointalk.org/index.php?topic=845978.0). It allows for compact proofs around the inclusion or exclusion of shares with particular namespace IDs. diff --git a/specs/src/specs/fraud_proofs.md b/specs/src/specs/fraud_proofs.md index 8dde0d33f5..2ec0802b00 100644 --- a/specs/src/specs/fraud_proofs.md +++ b/specs/src/specs/fraud_proofs.md @@ -22,4 +22,4 @@ State fraud proofs allow light clients to avoid making an honest majority assump state validity. While these are not incorporated into the protocol as of v1.0.0, there are example implementations that can be found in [Rollkit](https://github.com/rollkit/rollkit). More info in -[rollkit-ADR009](https://github.com/rollkit/rollkit/blob/4fd97ba8b8352771f2e66454099785d06fd0c31b/docs/lazy-adr/adr-009-state-fraud-proofs.md). \ No newline at end of file +[rollkit-ADR009](https://github.com/rollkit/rollkit/blob/4fd97ba8b8352771f2e66454099785d06fd0c31b/docs/lazy-adr/adr-009-state-fraud-proofs.md). diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index 6af6ccdc75..9f951d7e13 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -4,17 +4,17 @@ ## Abstract -One of Celestia's core data structures is the namespace. -When a user submits a transaction encapsulating a `MsgPayForBlobs` message to Celestia, they MUST associate each blob with exactly one namespace. +One of Celestia's core data structures is the namespace. +When a user submits a transaction encapsulating a `MsgPayForBlobs` message to Celestia, they MUST associate each blob with exactly one namespace. After their transaction has been included in a block, the namespace enables users to take an interest in a subset of the blobs published to Celestia by allowing the user to query for blobs by namespace. -In order to enable efficient retrieval of blobs by namespace, Celestia makes use of a [Namespaced Merkle Tree](https://github.com/celestiaorg/nmt). +In order to enable efficient retrieval of blobs by namespace, Celestia makes use of a [Namespaced Merkle Tree](https://github.com/celestiaorg/nmt). See section 5.2 of the [LazyLedger whitepaper](https://arxiv.org/pdf/1905.09274.pdf) for more details. ## Overview -A namespace is composed of two fields: [version](#version) and [id](#id). -A namespace is encoded as a byte slice with the version and id concatenated. +A namespace is composed of two fields: [version](#version) and [id](#id). +A namespace is encoded as a byte slice with the version and id concatenated. ![namespace](./figures/namespace.svg) @@ -23,17 +23,17 @@ A namespace is encoded as a byte slice with the version and id concatenated. The namespace version is an 8-bit unsigned integer that indicates the version of the namespace. The version is used to determine the format of the namespace and is encoded as a single byte. -[//]: # (TODO: It may be useful to indicate the endianness of the encoding) + A new namespace version MUST be introduced if the namespace format changes in a backwards incompatible way. - -Below we explain supported user-specifiable namespace versions, +Below we explain supported user-specifiable namespace versions, however, we note that Celestia MAY utilize other namespace versions for internal use. For more details, see the [Reserved Namespaces](#reserved-namespaces) section. #### Version 0 + The only supported user-specifiable namespace version is `0`. -A namespace with version `0` MUST contain an id with a prefix of 18 leading `0` bytes. +A namespace with version `0` MUST contain an id with a prefix of 18 leading `0` bytes. The remaining 10 bytes of the id are user-specified. Below, we provide examples of valid and invalid encoded user-supplied namespaces with version `0`. @@ -48,17 +48,19 @@ Below, we provide examples of valid and invalid encoded user-supplied namespaces 0x1000000000000000000000000000000000000000000000000000000000 // invalid because it does not have version 0 0x1111111111111111111111111111111111111111111111111111111111 // invalid because it does not have version 0 ``` + Any change in the number of leading `0` bytes in the id of a namespace with version `0` is considered a backwards incompatible change and MUST be introduced as a new namespace version. ### ID -The namespace ID is a 28 byte identifier that uniquely identifies a namespace. -The ID is encoded as a byte slice of length 28. -[//]: # (TODO: It may be useful to indicate the endianness of the encoding) +The namespace ID is a 28 byte identifier that uniquely identifies a namespace. +The ID is encoded as a byte slice of length 28. + ## Reserved Namespaces -Celestia reserves certain namespaces with specific meanings. -Applications SHOULD refrain from using these reserved namespaces for their blob data. [//]: # (TODO: the implication of this is still under investigation, but it is prudent to advise app developers not using this namespace) + +Celestia reserves certain namespaces with specific meanings. +Applications SHOULD refrain from using these reserved namespaces for their blob data. The rationale behind this is that these namespaces dictate the positioning of transactions and blobs data within the Celestia [data square](./data-square-layout.md). Therefore, by refraining from using these reserved namespaces, applications can guarantee the desired arrangement of their blob data. @@ -86,14 +88,14 @@ Applications SHOULD refrain from using the [reserved namespaces](#reserved-names See [pkg/namespace](../../../pkg/namespace). - ## Protobuf Definition [//]: # (TODO: Add protobuf definition for namespace if any) + ## References 1. [ADR-014](../../../docs/architecture/adr-014-versioned-namespaces.md) 1. [ADR-015](../../../docs/architecture/adr-015-namespace-id-size.md) 1. [Namespaced Merkle Tree](https://github.com/celestiaorg/nmt) 1. [LazyLedger whitepaper](https://arxiv.org/pdf/1905.09274.pdf) -1. [Data Square Layout](./data-square-layout.md) +1. [Data Square Layout](./data-square-layout.md) diff --git a/tools/blocktime/readme.md b/tools/blocktime/readme.md index e8d9ce4873..0ea48e9220 100644 --- a/tools/blocktime/readme.md +++ b/tools/blocktime/readme.md @@ -8,13 +8,13 @@ To read up on starting a node and exposing the RPC endpoint go to the docs [here To compile the binary, run either `go install` or `go build`. The binary can then be used as follows: -``` -$ ./blocktime [query_range] +```bash +./blocktime [query_range] ``` As an example -``` +```bash $ ./blocktime http://localhost:26657 1000 Chain: mocha-3 From 912b9411f7c7d11fc68f55a7c7448e479a1c2d8a Mon Sep 17 00:00:00 2001 From: sanaz Date: Tue, 18 Jul 2023 13:19:03 -0700 Subject: [PATCH 13/20] removes a comment --- specs/src/specs/namespace.md | 1 - 1 file changed, 1 deletion(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index 9f951d7e13..2c184d86cb 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -39,7 +39,6 @@ Below, we provide examples of valid and invalid encoded user-supplied namespaces ```go // Valid encoded namespaces -0x0000000000000000000000000000000000000000000000000000000001 // transaction namespace [?] is this user-specified? I mean, when sending a transaction, does a user need to associate this namespace before submission of the tx to a validator? or is this namespace added later during the data square construction? 0x0000000000000000000000000000000000000001010101010101010101 // valid blob namespace 0x0000000000000000000000000000000000000011111111111111111111 // valid blob namespace From c49fd2f3b06f7623d1ef6a03624d74f7effbc060 Mon Sep 17 00:00:00 2001 From: sanaz Date: Tue, 18 Jul 2023 13:22:52 -0700 Subject: [PATCH 14/20] adds todo for protobuf --- specs/src/specs/namespace.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index 2c184d86cb..48f70c2aae 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -89,7 +89,7 @@ See [pkg/namespace](../../../pkg/namespace). ## Protobuf Definition -[//]: # (TODO: Add protobuf definition for namespace if any) + ## References From c901967d4d63111910ec7e4b8a9b193520595e5b Mon Sep 17 00:00:00 2001 From: sanaz Date: Tue, 18 Jul 2023 15:18:46 -0700 Subject: [PATCH 15/20] fixes a dead link --- specs/src/specs/namespace.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index 48f70c2aae..7cc67e01ec 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -60,14 +60,14 @@ The ID is encoded as a byte slice of length 28. Celestia reserves certain namespaces with specific meanings. Applications SHOULD refrain from using these reserved namespaces for their blob data. -The rationale behind this is that these namespaces dictate the positioning of transactions and blobs data within the Celestia [data square](./data-square-layout.md). +The rationale behind this is that these namespaces dictate the positioning of transactions and blobs data within the Celestia [data square](./data_square_layout.md). Therefore, by refraining from using these reserved namespaces, applications can guarantee the desired arrangement of their blob data. Below is a list of reserved namespaces, along with a brief description of each. In the table, you will notice that the `PARITY_SHARE_NAMESPACE` and `TAIL_PADDING_NAMESPACE` utilize the namespace version `255`, which differs from the supported user-specified versions. The reason for employing version `255` for the `PARITY_SHARE_NAMESPACE` is to enable more efficient proof generation within the context of [nmt](https://github.com/celestiaorg/nmt), where it is used in conjunction with the `IgnoreMaxNamespace` feature. Similarly, the `TAIL_PADDING_NAMESPACE` utilizes the namespace version `255` to ensure that padding shares are always properly ordered and placed at the end of the Celestia data square even if a new namespace version is introduced. -For additional information on the significance and application of the reserved namespaces, please refer to the [Data Square Layout](./data-square-layout.md) section. +For additional information on the significance and application of the reserved namespaces, please refer to the [Data Square Layout](./data_square_layout.md) section. | name | type | value | description | |-------------------------------------|-------------|----------------------------------------------------------------|------------------------------------------------------------------------------------------------------| @@ -97,4 +97,4 @@ See [pkg/namespace](../../../pkg/namespace). 1. [ADR-015](../../../docs/architecture/adr-015-namespace-id-size.md) 1. [Namespaced Merkle Tree](https://github.com/celestiaorg/nmt) 1. [LazyLedger whitepaper](https://arxiv.org/pdf/1905.09274.pdf) -1. [Data Square Layout](./data-square-layout.md) +1. [Data Square Layout](./data_square_layout.md) From 05c587d8bcf4d09130855041c4e8491ab50b81d0 Mon Sep 17 00:00:00 2001 From: sanaz Date: Tue, 18 Jul 2023 15:24:41 -0700 Subject: [PATCH 16/20] a minor rewording --- specs/src/specs/namespace.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index 7cc67e01ec..ff2990ecc4 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -67,7 +67,7 @@ Below is a list of reserved namespaces, along with a brief description of each. In the table, you will notice that the `PARITY_SHARE_NAMESPACE` and `TAIL_PADDING_NAMESPACE` utilize the namespace version `255`, which differs from the supported user-specified versions. The reason for employing version `255` for the `PARITY_SHARE_NAMESPACE` is to enable more efficient proof generation within the context of [nmt](https://github.com/celestiaorg/nmt), where it is used in conjunction with the `IgnoreMaxNamespace` feature. Similarly, the `TAIL_PADDING_NAMESPACE` utilizes the namespace version `255` to ensure that padding shares are always properly ordered and placed at the end of the Celestia data square even if a new namespace version is introduced. -For additional information on the significance and application of the reserved namespaces, please refer to the [Data Square Layout](./data_square_layout.md) section. +For additional information on the significance and application of the reserved namespaces, please refer to the [Data Square Layout](./data_square_layout.md) specifications. | name | type | value | description | |-------------------------------------|-------------|----------------------------------------------------------------|------------------------------------------------------------------------------------------------------| From f48db891ca9883db17dfc8285117366377194228 Mon Sep 17 00:00:00 2001 From: sanaz Date: Wed, 19 Jul 2023 10:02:03 -0700 Subject: [PATCH 17/20] removes a comment about endianness of version value --- specs/src/specs/namespace.md | 1 - 1 file changed, 1 deletion(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index ff2990ecc4..e9b517db73 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -23,7 +23,6 @@ A namespace is encoded as a byte slice with the version and id concatenated. The namespace version is an 8-bit unsigned integer that indicates the version of the namespace. The version is used to determine the format of the namespace and is encoded as a single byte. - A new namespace version MUST be introduced if the namespace format changes in a backwards incompatible way. Below we explain supported user-specifiable namespace versions, From 09ed3bbc159919de94bf253bd08f7bb688d17e7c Mon Sep 17 00:00:00 2001 From: sanaz Date: Wed, 19 Jul 2023 10:24:52 -0700 Subject: [PATCH 18/20] revises the reserved namespaces intro paragraph --- specs/src/specs/namespace.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index e9b517db73..053a273a4e 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -58,9 +58,8 @@ The ID is encoded as a byte slice of length 28. ## Reserved Namespaces Celestia reserves certain namespaces with specific meanings. +Celestia makes use of the reserved namespaces to properly organize and order transactions and blobs inside the [data square](./data_square_layout.md). Applications SHOULD refrain from using these reserved namespaces for their blob data. -The rationale behind this is that these namespaces dictate the positioning of transactions and blobs data within the Celestia [data square](./data_square_layout.md). -Therefore, by refraining from using these reserved namespaces, applications can guarantee the desired arrangement of their blob data. Below is a list of reserved namespaces, along with a brief description of each. In the table, you will notice that the `PARITY_SHARE_NAMESPACE` and `TAIL_PADDING_NAMESPACE` utilize the namespace version `255`, which differs from the supported user-specified versions. From f171508ce374deda53303124ca7f6b3235498fed Mon Sep 17 00:00:00 2001 From: sanaz Date: Wed, 19 Jul 2023 10:44:09 -0700 Subject: [PATCH 19/20] replaces SHOULD with MUST --- specs/src/specs/namespace.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index 053a273a4e..f33b0b261c 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -59,7 +59,7 @@ The ID is encoded as a byte slice of length 28. Celestia reserves certain namespaces with specific meanings. Celestia makes use of the reserved namespaces to properly organize and order transactions and blobs inside the [data square](./data_square_layout.md). -Applications SHOULD refrain from using these reserved namespaces for their blob data. +Applications MUST NOT use these reserved namespaces for their blob data. Below is a list of reserved namespaces, along with a brief description of each. In the table, you will notice that the `PARITY_SHARE_NAMESPACE` and `TAIL_PADDING_NAMESPACE` utilize the namespace version `255`, which differs from the supported user-specified versions. @@ -79,7 +79,7 @@ For additional information on the significance and application of the reserved n ## Assumptions and Considerations -Applications SHOULD refrain from using the [reserved namespaces](#reserved-namespaces) for their blob data. +Applications MUST refrain from using the [reserved namespaces](#reserved-namespaces) for their blob data. ## Implementation From d39223bcc17b07d87335ebbce99b7ea8958d4473 Mon Sep 17 00:00:00 2001 From: sanaz Date: Wed, 19 Jul 2023 10:49:11 -0700 Subject: [PATCH 20/20] removes a stale comment --- specs/src/specs/namespace.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/src/specs/namespace.md b/specs/src/specs/namespace.md index f33b0b261c..43d2506e7f 100644 --- a/specs/src/specs/namespace.md +++ b/specs/src/specs/namespace.md @@ -59,7 +59,7 @@ The ID is encoded as a byte slice of length 28. Celestia reserves certain namespaces with specific meanings. Celestia makes use of the reserved namespaces to properly organize and order transactions and blobs inside the [data square](./data_square_layout.md). -Applications MUST NOT use these reserved namespaces for their blob data. +Applications MUST NOT use these reserved namespaces for their blob data. Below is a list of reserved namespaces, along with a brief description of each. In the table, you will notice that the `PARITY_SHARE_NAMESPACE` and `TAIL_PADDING_NAMESPACE` utilize the namespace version `255`, which differs from the supported user-specified versions.