diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 92c0a26..df2e5bd 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -1,18 +1,42 @@ # Summary -- [Introduction](./introduction.md) +[Introduction](./introduction.md) + +# Guides + +- [Architecture](./architecture/index.md) + - [Polka Storage Provider Server](./architecture/polka-storage-provider-server.md) + - [Pallets](./architecture/pallets/index.md) + - [Market](./architecture/pallets/market.md) + - [Storage Provider](./architecture/pallets/storage-provider.md) + - [Proofs](./architecture/pallets/proofs.md) + - [Randomness](./architecture/pallets/randomness.md) - [Getting Started](./getting-started/index.md) - - [Local Testnet - Polka Storage Parachain](./getting-started/local-testnet.md) - - [CAR server](./getting-started/car-server.md) - - [Real-world use case demo](./getting-started/demo.md) + - [Building](./getting-started/building/index.md) + - [From Source](./getting-started/building/source.md) + - [Using Docker](./getting-started/building/docker.md) + - [Local Testnet - Polka Storage Parachain](./getting-started/local-testnet/index.md) + - [Getting funds](./getting-started/local-testnet/getting-funds.md) + - [Launching the Storage Provider](./getting-started/storage-provider.md) + - [Proving a file](./getting-started/demo-file-prove.md) + - [Storing a file](./getting-started/demo-file-store.md) + +# Tools + - [Polka Storage Provider CLI](./storage-provider-cli/index.md) - - [`storage`](./storage-provider-cli/storage.md) - - [`wallet`](./storage-provider-cli/wallet.md) + - [`server`](./storage-provider-cli/server.md) + - [`client`](./storage-provider-cli/client/index.md) + - [`wallet`](./storage-provider-cli/client/wallet.md) + - [`proofs`](./storage-provider-cli/client/proofs.md) - [Storagext CLI](./storagext-cli/index.md) - [`market`](./storagext-cli/market.md) - [`storage-provider`](./storagext-cli/storage-provider.md) + - [`proofs`](./storagext-cli/proofs.md) + - [`randomness`](./storagext-cli/randomness.md) - [`system`](./storagext-cli/system.md) -- [Pallets](./pallets/index.md) - - [Market](./pallets/market.md) - - [Storage Provider](./pallets/storage-provider.md) -- [Glossary](./glossary.md) +- [Mater CLI](./mater-cli/index.md) +- [Zombienet Configuration](./zombienet-config.md) + +--- + +[Glossary](./glossary.md) diff --git a/src/architecture/index.md b/src/architecture/index.md new file mode 100644 index 0000000..8f5e446 --- /dev/null +++ b/src/architecture/index.md @@ -0,0 +1,45 @@ +# Architecture Overview + +The Polka Storage parachain is, just like other parachains, composed of collators that receive extrinsics calls, +and through them perform state transitions. + +## System Overview + +![](../images/architecture/system_overview.png) + +From left to right, we have validators (represented by a single node as only one validates blocks at a time), +collators and the storage providers. + +The validators are handled by Polkadot itself — i.e. who gets to check the validity proofs is randomly selected by the network. + +The collators execute our parachain runtime and process extrinsic calls from the storage providers — such as proof of storage submissions. + +The storage providers are independent of the collators and are controlled by people like you, who provide storage to the system. +Storage management is left to the storage providers, being responsible to keep their physical system in good shape to serve clients. +We do provide an implementation of the storage provider, you can read more about it in the [Polka Storage Provider Server chapter](./polka-storage-provider-server.md). + +## Pallets Overview + +![](../images/architecture/pallets_overview.png) + +We've been focusing on implementing the core functionality by developing the market, storage provider, proofs and randomness pallets. + +The market pallet handles all things related to deal payments and slashing, +being informed by the storage provider when deals haven't been proven and applying slashing in those cases. +The storage provider handles the registering of storage providers and the proof submission, +the latter is checked inside the collator's WASM runtime, using the proofs pallet. +Finally, the proofs pallet makes use of randomness for challenges, ensuring the storage providers can't cheat the system. + +For a deeper dive on the pallets, you can read the [Pallets chapter](./pallets/index.md). + +## Resources on Parachains + +Reading: +* [Parachains' Protocol Overview](https://wiki.polkadot.network/docs/learn-parachains-protocol) +* [The Path of a Parachain Block](https://polkadot.com/blog/the-path-of-a-parachain-block) + +Videos: +* [Introduction to Polkadot, Parachains, and Substrate](https://www.youtube.com/live/gT-9r1bcVHY?si=dmCJyWB5w2NY1bnu&t=1670) +* [The Path of a Parachain Block - Joe Petrowski](https://www.youtube.com/watch?v=vRsBlVELQEo) +* [The Path of a Parachain Block on Polkadot and Kusama Network](https://www.youtube.com/watch?v=m0vxqWwFfDs) + diff --git a/src/architecture/pallets/index.md b/src/architecture/pallets/index.md new file mode 100644 index 0000000..30ff44e --- /dev/null +++ b/src/architecture/pallets/index.md @@ -0,0 +1,55 @@ +# Polka Storage pallets + +- [`storage-provider`](storage-provider.md) - A pallet that manages storage providers and their associated data. +- [`market`](market.md) - A pallet that handles the storage market operations. +- [`proofs`](proofs.md) - A pallet responsible for verifying [PoRep](../../glossary.md#porep) and [PoSt](../../glossary.md#post). +- [`randomness`](randomness.md) - A pallet providing randomness source for blocks, mainly used by Proofs. + +## Overview + +The Polka Storage parachain is all about making storage deals. Let us go over how a deal is done! + +Before anything else, [Storage Providers](../../glossary.md#storage-provider) need to register themselves with the [Storage Provider Pallet](./storage-provider.md) — they can do so using the [`register_storage_provider`](./storage-provider.md#register_storage_provider) extrinsic. + +Storage Provider registration + +Now that storage providers can be registered in the storage provider pallet, we need to add some balance to both the +[Storage User's](../../glossary.md#storage-user) and the Provider's accounts, +which is done using the Market's [`add_balance`](./market.md#add_balance) extrinsic. + +Adding balance to Market accounts + +Afterwards, storage users and providers negotiate data storage deals off-chain. +Once a deal between the two parties is reached, the client can sign the deal and send it to the storage provider for publishing +— the storage provider will then publish the signed deal using the [`publish_storage_deals`](market.md#publish_storage_deals) extrinsic. + +After publishing, the funds allocated for the deal will be moved from `free` to `locked`, and they can no longer be withdrawn until the deal has ended. + +Publishing storage deals + +At this point, the remaining responsibility is shifted to the storage provider, which needs to activate the deal. +First, the storage provider needs to call `get_randomness` from the [Randomness Pallet](./randomness.md) in order to create a replica and +[pre-commit](./storage-provider.md#pre_commit_sectors) the deal's sectors. +The sealing and pre-committing takes some time, after that the storage provider needs to fetch yet another randomness seed to create a proof. +Subsequently, they [prove](./storage-provider.md#prove_commit_sectors) they stored the sectors by calling [`prove_commit_sectors`](./storage-provider.md#prove_commit_sectors) extrinsics. + +Verification is done via the [Proofs Pallet](./proofs.md) and reported to the Market pallet to terminate the deal and apply penalties to the storage provider +(remove and burn its collateral — i.e. `locked` funds) if they fail to activate the deal on time and return the funds to the client. + +Deal activation + +Suppose the deal has been completed successfully or is **Active**. +In that case, the storage provider is now required to periodically submit proofs that they're still storing the user's data +— the storage provider does this by calculating a proof and submitting it using [`submit_windowed_post`](./storage-provider.md#submit_windowed_post). + +Proving the data is still stored + +Finally, storage providers can then settle deal payments to receive their fair share for keeping the user's data safe — using the [`settle_deal_payments`](./market.md#settle_deal_payments) extrinsic. + +Settling deal payments + +Putting it all together, we get the following: + +The described flow + + diff --git a/src/pallets/market.md b/src/architecture/pallets/market.md similarity index 86% rename from src/pallets/market.md rename to src/architecture/pallets/market.md index 1439c57..504db7a 100644 --- a/src/pallets/market.md +++ b/src/architecture/pallets/market.md @@ -18,7 +18,7 @@ The purpose of the pallet is to manage storage deals between storage market part Market Pallet is tightly coupled with [Storage Provider Pallet](./storage-provider.md) because it's a source of truth for deals. Storage Provider Pallet cannot exist without deal information from Market Pallet. -## Extrinsics\* +## Extrinsics\* ### `add_balance` @@ -32,7 +32,7 @@ The reserved amount will be considered `free` until it is used in a deal when #### Example -Using the `storagext-cli` to add 1000000000[^existential_deposit] [Plancks](../glossary.md#planck) to Alice's account with the following command[^add_balance]: +Using the `storagext-cli` to add 1_000_000_000[^existential_deposit] [Plancks](../../glossary.md#planck) to Alice's account with the following command[^add_balance]: ```bash storagext-cli --sr25519-key "//Alice" market add-balance 1000000000 @@ -42,14 +42,14 @@ storagext-cli --sr25519-key "//Alice" market add-balance 1000000000 This value is the minimum amount due to Polkadot's existential deposit. More information available in: . -[^add_balance]: Read more about the `add-balance` command in [_Storagext CLI/Subcommand `market`/`add-balance`_](../storagext-cli/market.md#add-balance) +[^add_balance]: Read more about the `add-balance` command in [_Storagext CLI/Subcommand `market`/`add-balance`_](../../storagext-cli/market.md#add-balance). ### `withdraw_balance` Withdraws funds from the Storage Market. The funds will be withdrawn from the `free` balance, meaning that the `amount` must be -lesser than or equal to `free` and greater than 0 (\\({free} \ge {amount} \gt 0\\)). +less than or equal to `free` and greater than 0 (\\({free} \ge {amount} \gt 0\\)). | Name | Description | Type | | -------- | -------------------------- | ---------------- | @@ -57,13 +57,13 @@ lesser than or equal to `free` and greater than 0 (\\({free} \ge {amount} \gt 0\ #### Example -Using the `storagext-cli` to withdraw 10000 [Plancks](../glossary.md#planck) from Alice's `free` balance using the following command[^withdraw_balance]: +Using the `storagext-cli` to withdraw 10000 [Plancks](../../glossary.md#planck) from Alice's `free` balance using the following command[^withdraw_balance]: ```bash storagext-cli --sr25519-key "//Alice" market withdraw-balance 10000 ``` -[^withdraw_balance]: Read more about the `withdraw-balance` command in [_Storagext CLI/Subcommand `market`/`withdraw-balance`_](../storagext-cli/market.md#withdraw-balance) +[^withdraw_balance]: Read more about the `withdraw-balance` command in [_Storagext CLI/Subcommand `market`/`withdraw-balance`_](../../storagext-cli/market.md#withdraw-balance). ### `publish_storage_deals` @@ -73,11 +73,11 @@ This extrinsic _must_ be called by a storage provider. | Name | Description | Type | | ------------------ | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `proposal` | Specific deal proposal, a JSON object | JSON object, specified in the [next section](#deal-proposal-components) | +| `proposal` | Specific deal proposal, a JSON object | JSON object, specified in the [deal proposal components section](#deal-proposal-components) | | `client_signature` | Client signature of this specific deal proposal | [`MultiSignature`](https://paritytech.github.io/polkadot-sdk/master/sp_runtime/enum.MultiSignature.html), meaning a 64-byte array for Sr25519 and Ed25519 signatures and 65-byte array for ECDSA signatures | The `client_signature`, as the name indicates, is generated by the client by signing the deal proposal with their private key — -the [`storagext-cli`](../storagext-cli/index.md) does this for the user automatically[^warn]. +the [`storagext-cli`](../../storagext-cli/index.md) does this for the user automatically[^warn]. This signature ensures that the storage provider cannot forge a deal with an arbitrary client. The type of signature is dependent on the key the signer has, currently supported key types are Sr25519, ECDSA and Ed25519. @@ -94,8 +94,8 @@ _This step corresponds to the "sign & send proposal" step in the [deal overview] | `label` | Arbitrary client chosen label | String, with a maximum length of 128 characters | | `start_block` | Block number on which the deal should start | Positive integer | | `end_block` | Block number on which the deal should end | Positive integer, `end_block > start_block` | -| `storage_price_per_block` | Price for the storage specified per block[^storage_price_per_block] | Positive integer, in [Plancks](../glossary.md#planck) | -| `provider_collateral` | Collateral which is slashed if the deal fails | Positive integer, in [Plancks](../glossary.md#planck) | +| `storage_price_per_block` | Price for the storage specified per block[^storage_price_per_block] | Positive integer, in [Plancks](../../glossary.md#planck) | +| `provider_collateral` | Collateral which is slashed if the deal fails | Positive integer, in [Plancks](../../glossary.md#planck) | | `state` | Deal state. Can only be set to `Published` | String | See the [original Filecoin specification](https://spec.filecoin.io/#section-systems.filecoin_markets.onchain_storage_market.storage_deal_flow) for details. @@ -149,9 +149,9 @@ and is **not final**; it is a testing tool. -[^storage_price_per_block]: The formula to calculate the total price is as follows: \\[total\\\_price = (end\\\_block - start\\\_block) * storage\\_price\\_per\\_block\\] -[^publish_storage_deals]: Read more about the `publish-storage-deals` command in [_Storagext CLI/Subcommand `market`/`publish-storage-deals`_](../storagext-cli/market.md#publish-storage-deals) [^warn]: Take into account that the CLI is currently for demo purposes, the authors are aware that the command isn't safe since it requires the private keys of both parties. +[^storage_price_per_block]: The formula to calculate the total price is as follows: \\\[total\\\_price = (end\\\_block - start\\\_block) \* storage\\\_price\\\_per\\\_block\\]. +[^publish_storage_deals]: Read more about the `publish-storage-deals` command in [_Storagext CLI/Subcommand `market`/`publish-storage-deals`_](../../storagext-cli/market.md#publish-storage-deals). ### `settle_deal_payments` @@ -174,7 +174,7 @@ Using the `storagext-cli` to settle deal payments for IDs 97, 1010, 1337 and 420 storagext-cli --sr25519-key "//Alice" market settle-deal-payments 97 1010 1337 42069 ``` -[^settle_deal_payments]: Read more about the `settle-deal-payments` command in [_Storagext CLI/Subcommand `market`/`settle-deal-payments`_](../storagext-cli/market.md#settle-deal-payments) +[^settle_deal_payments]: Read more about the `settle-deal-payments` command in [_Storagext CLI/Subcommand `market`/`settle-deal-payments`_](../../storagext-cli/market.md#settle-deal-payments) ## Events @@ -197,9 +197,9 @@ The Market Pallet emits the following events: - `DealsSettled` - Published after the `settle_deal_payments` extrinsic is called. Indicates which deals were successfully and unsuccessfully settled. - `successful` - List of deal IDs that were settled - `unsuccessful` - List of deal IDs with the corresponding errors -- `DealSlashed` - Is emitted when some deal expired +- `DealSlashed` - Is emitted when some deal expired. - `deal_id` - Deal ID that was slashed -- `DealTerminated` - Is emitted it indicates that the deal was voluntarily or involuntarily terminated. +- `DealTerminated` - A deal was voluntarily or involuntarily terminated. - `deal_id` - Terminated deal ID - `client` - SS58 address of the storage client - `provider` - SS58 address of the storage provider @@ -210,7 +210,7 @@ The Market Pallet actions can fail with following errors: - `InsufficientFreeFunds` - Market participants do not have enough free funds. - `NoProposalsToBePublished` - `publish_storage_deals` was called with an empty list of `deals`. -- `ProposalsNotPublishedByStorageProvider` - Is returned when calling `publish_storage_deals` and the deals in a list are not published by the same storage provider. +- `ProposalsPublishedByIncorrectStorageProvider` - Is returned when calling `publish_storage_deals` and the deals in a list are not published by the same storage provider. - `AllProposalsInvalid` - `publish_storage_deals` call was supplied with a list of `deals` which are all invalid. - `DuplicateDeal` - There is more than one deal with this ID in the Sector. - `DealNotFound` - Tried to activate a deal that is not in the system. @@ -227,9 +227,9 @@ The Market Pallet actions can fail with following errors: ## Constants -| Name | Description | Value | -| ------------------ | ----------------------------------------------------------------------------- | ----------- | -| `MaxDeals` | How many deals can be published in a single batch of `publish_storage_deals`. | 128 | -| `MaxDealsPerBlock` | Maximum deals that can be scheduled to start at the same block. | 128 | -| `MinDealDuration` | Minimum time an activated deal should last. | 5 Minutes | -| `MaxDealDuration` | Maximum time an activated deal should last. | 180 Minutes | +| Name | Description | Value | +| ------------------ | ----------------------------------------------------------------------------- | ------------------------- | +| `MaxDeals` | How many deals can be published in a single batch of `publish_storage_deals`. | 128 | +| `MaxDealsPerBlock` | Maximum deals that can be scheduled to start at the same block. | 128 | +| `MinDealDuration` | Minimum time an activated deal should last. | 5 Minutes (50 Blocks) | +| `MaxDealDuration` | Maximum time an activated deal should last. | 180 Minutes (1800 Blocks) | diff --git a/src/architecture/pallets/proofs.md b/src/architecture/pallets/proofs.md new file mode 100644 index 0000000..cc3ed2c --- /dev/null +++ b/src/architecture/pallets/proofs.md @@ -0,0 +1,93 @@ +# Proofs Pallet + +## Table of Contents + +- [Overview](#overview) +- [Usage](#usage) +- [Extrinsics](#extrinsics) + - [`set_porep_verifying_key`](#set_porep_verifying_key) + - [`set_post_verifying_key`](#set_post_verifying_key) +- [Events](#events) +- [Errors](#errors) + +## Overview + +The `Proofs Pallet` handles all the logic related to verifying [PoRep](../../glossary.md#porep) and [PoSt](../../glossary.md#post) proofs on-chain. +It's called by [`Storage Provider Pallet`](./storage-provider.md) when verifying proofs during the extrinsics [`prove_commit_sectors`](./storage-provider.md#prove_commit_sectors) +and [`submit_windowed_post`](./storage-provider.md#submit_windowed_post). The Pallet **DOES NOT** expose any extrinsic for proofs verification, it only implements a trait that can be [coupled to other pallets](https://education.web3.foundation/docs/Substrate/section8/pallet-coupling). + +To verify the proofs properly it needs to have the verifying keys parameters set for the sector size via [`set_porep_verifying_key`](#set_porep_verifying_key) and [`set_post_verifying_key`](#set_post_verifying_key). + +## Usage + +This pallet can only be directly used via the trait `primitives_proofs::ProofVerification`. However, for the trait to work and not fail with `Error::MisingPoRepVerifyingKey`/`Error::MissingPoStVerifingKey`, the verifying keys need to be set via extrinsics `set_porep_verifying_key`/`set_post_verifying_key`. + +Ideally, users shouldn't worry about it, as it will be set by the governance during a trusted setup procedure and then Storage Providers will download the proof generation parameters. However, in the MVP phase, those keys need to be set with the extrinsics after starting a testnet. + +Verifying Keys are set for a Sector Size once and then shared across all proof verifications. +Currently, the network only supports 2KiB sector sizes, so parameters need to be generated and set for it. + +## Extrinsics + +### `set_porep_verifying_key` + +Verifying Key is a set of shared parameters used for zk-SNARK proof verification. It can be generated via [`polka-storage-provider-client proofs porep-params`](../../storage-provider-cli/client/proofs.md#porep-params) command. The verifying key used in the verification must match the proving parameters used in the proof generation. + +The extrinsic sets the verifying key received in the SCALE-encoded format and then uses it for all the subsequent verification. +Verifying Key is used to verify every PoRep proof across the network. + +| Name | Description | Type | +| --------------- | ------------------------------------------------------------- | -------------------------------------- | +| `verifying_key` | shared set of parameters used for zk-SNARK proof verification | SCALE encoded bytes of a Verifying Key | + +#### Example + +Setting a verifying key from the [^account] `//Alice` account where proof is stored in the `./2KiB.porep.vk.scale` file. + +```bash +storagext-cli --sr25519-key "//Alice" proofs set-porep-verifying-key 2KiB.vk.scale +``` + +[^account]: Note that in the MVP every account can set a Verifying Key. It's a risky operation that can halt the entire network, because if verifying key changes, Storage Providers needs to update their generating parameters as well. + +### `set_post_verifying_key` + +Verifying Key is a set of shared parameters used for zk-SNARK proof verification. It can be generated via [`polka-storage-provider-client proofs post-params`](../../storage-provider-cli/client/proofs.md#post-params) command. The verifying key used in the verification must match proving parameters used in the proof generation. + +The extrinsic sets the verifying key received in the SCALE-encoded format and then uses it for all the subsequent verification. +Verifying Key is used to verify every PoSt proof across the network. + +| Name | Description | Type | +| --------------- | ------------------------------------------------------------- | -------------------------------------- | +| `verifying_key` | shared set of parameters used for zk-SNARK proof verification | SCALE encoded bytes of a Verifying Key | + +#### Example + +Setting a verifying key from the [^account] `//Alice` account where proof is stored in the `./2KiB.post.vk.scale` file. + +```bash +storagext-cli --sr25519-key "//Alice" proofs set-post-verifying-key 2KiB.vk.scale +``` + +[^account]: Note that in the MVP every account can set a Verifying Key. It's a risky operation that can halt the entire network, because if verifying key changes, Storage Providers needs to update their generating parameters as well. + +## Events + +The Proofs Pallet emits the following events: + +- `PoRepVerifyingKeyChanged` - PoRep verifying key has been changed. + - `who` - SS58 address of the caller. +- `PoStVerifyingKeyChanged` - PoSt verifying key has been changed. + - `who` - SS58 address of the caller. + +## Errors + +The Proofs Pallet actions can fail with the following errors: + +- `InvalidVerifyingKey` - supplied Verifying Key was not in the valid format and could not be deserialized. +- `InvalidPoRepProof` - PoRep proof could not be verified, it was not created for the given replica window. +- `InvalidPoStProof` - PoSt proof could not be verified, it was not created for the given sector. +- `MissingPoRepVerifyingKey` - tried to verify PoRep proof, but the PoRep verifying key was not set previously with the [`set_porep_verifying_key`](#set_post_verifying_key) extrinsic. +- `MissingPoStVerifyingKey` - tried to verify PoSt proof, but the PoSt verifying key was not set previously with the [`set_post_verifying_key`](#set_post_verifying_key) extrinsic. +- `Conversion` - PoRep/PoSt Proof/VerifyingKey are in an invalid format and cannot be deserialized. + diff --git a/src/architecture/pallets/randomness.md b/src/architecture/pallets/randomness.md new file mode 100644 index 0000000..9dd5d28 --- /dev/null +++ b/src/architecture/pallets/randomness.md @@ -0,0 +1,48 @@ +# Randomness Pallet + +## Table of Contents + +- [Randomness Pallet](#randomness-pallet) + - [Table of Contents](#table-of-contents) + - [Overview](#overview) + - [Usage](#usage) + - [Extrinsics](#extrinsics) + - [Events](#events) + - [Errors](#errors) + - [Constants](#constants) + +## Overview + +The randomness pallet saves a random seed for each block when it's finalized and allows retrieval of this randomness at a later time. +There is a limitation - the randomness is available only after the 81st block of the chain, due to randomness predictability earlier. +Currently, the seeds are used for the sealing pipeline's pre-commit and prove commit, in other words generating a replica and proving a sector. + +## Usage + +This pallet exposes the interface to get randomness on-chain for a certain block via the trait `primitives_proofs::Randomness` +or chain state query `pallet_randomness:SeedsMap`. +Note that, you can only get a randomness for the `current_block - 1` and depending on the configuration, the old randomness seed will be removed after the associated block has passed. + +## Extrinsics + +The pallet does not expose any extrinsics. + +## Events + +The pallet does not emit any events. + +## Errors + +The Randomness Pallet actions can fail with the following errors: + +- `SeedNotAvailable` - the seed for the given block number is not available, which means the randomness pallet has not gathered randomness for this block yet. + +## Constants + +The Storage Provider Pallet has the following constants: + +| Name | Description | Value | +| ----------------- | ----------------------------------------------------------------- | ------- | +| `CleanupInterval` | Clean-up interval specified in number of blocks between cleanups. | 1 Day | +| `SeedAgeLimit` | The number of blocks after which the seed is cleaned up. | 30 Days | + diff --git a/src/pallets/storage-provider-extra.md b/src/architecture/pallets/storage-provider-extra.md similarity index 100% rename from src/pallets/storage-provider-extra.md rename to src/architecture/pallets/storage-provider-extra.md diff --git a/src/pallets/storage-provider.md b/src/architecture/pallets/storage-provider.md similarity index 61% rename from src/pallets/storage-provider.md rename to src/architecture/pallets/storage-provider.md index 7950e6d..0ee5971 100644 --- a/src/pallets/storage-provider.md +++ b/src/architecture/pallets/storage-provider.md @@ -7,11 +7,12 @@ - [Declaring storage faults and recoveries](#declaring-storage-faults-and-recoveries) - [Extrinsics](#extrinsics) - [`register_storage_provider`](#register_storage_provider) - - [`pre_commit_sector`](#pre_commit_sector) - - [`prove_commit_sector`](#prove_commit_sector) + - [`pre_commit_sectors`](#pre_commit_sectors) + - [`prove_commit_sectors`](#prove_commit_sectors) - [`submit_windowed_post`](#submit_windowed_post) - [`declare_faults`](#declare_faults) - [`declare_faults_recovered`](#declare_faults_recovered) + - [`terminate_sectors`](#terminate_sectors) - [Events](#events) - [Errors](#errors) - [Pallet constants](#pallet-constants) @@ -19,26 +20,24 @@ ## Overview The `Storage Provider Pallet` handles the creation of storage providers and facilitates storage providers and clients in creating storage deals. -Storage providers must provide the [Proof of Space-time (PoSt)](../glossary.md#post) and the [Proof of Replication (PoRep)](../glossary.md#porep) -to the `Storage Provider Pallet` to prevent the pallet from imposing penalties on storage providers through [slashing](#storage-fault-slashing). +Storage providers must provide the [Proof of Space-time (PoSt)](../../glossary.md#post) and the [Proof of Replication (PoRep)](../../glossary.md#porep) +to the `Storage Provider Pallet` to prevent the pallet from imposing penalties on storage providers through [slashing](../../glossary.md#slashing). ## Usage ### Declaring storage faults and recoveries Faulty sectors are subject to penalties. To minimize said penalties, the storage provider should declare any sector for which they cannot -generate a [PoSt](../glossary.md#post) as faulty, this will mask said sectors in future deadlines, minimizing the suffered penalties. +generate a [PoSt](../../glossary.md#post) as faulty, this will mask said sectors in future deadlines, minimizing the suffered penalties. A storage provider must declare the sector as faulty [**before**](#fault-declaration-cutoff) the challenge window. Through the [`declare_faults`](#declare_faults) and [`declare_faults_recovered`](#declare_faults_recovered) extrinsics the storage provider can declare sectors as faulty or recovered[^recovered]. -Declaring faults and recoveries +Declaring faults and recoveries [^recovered]: Recovered sectors still require being proven before they can become fully active again. -Substrate pallet hooks execute actions when certain conditions are met. - Substrate pallet hooks execute some actions when certain conditions are met. We use these hooks — when a block finalizes — to check if storage providers are up to date with their proofs. If a storage provider fails to submit proof on time, the Storage Provider pallet will signal the Market pallet to penalize the storage provider. @@ -65,21 +64,22 @@ Registering a storage provider with keypair `//Alice` and peer ID `alice` with t storagext-cli --sr25519-key "//Alice" storage-provider register alice ``` -[^register_storage_provider]: Read more about the `register` command in [_Storagext CLI/Subcommand `storage-provider`/`register`_](../storagext-cli/storage-provider.md#register) +[^register_storage_provider]: Read more about the `register` command in [_Storagext CLI/Subcommand `storage-provider`/`register`_](../../storagext-cli/storage-provider.md#register) -### `pre_commit_sector` +### `pre_commit_sectors` After publishing a deal, the storage provider needs to pre-commit the sector information to the chain. Sectors are not valid after pre-commit. The sectors need to be proven first. +The pre-commit extrinsic takes in an array of the following values: -| Name | Description | Type | -| --------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------- | -| `seal_proof` | Seal proof type this storage provider is using [^note] | String, currently only `StackedDRGWindow2KiBV1P1` is available | -| `sector_number` | The sector number that is being pre-committed | Positive integer | -| `sealed_cid` | [Commitment of replication](../glossary.md#commitment-of-replication) | Hex string of the sealed CID bytes | -| `deal_ids` | Deal IDs to be pre-committed, from `publish_storage_deals` | Array of integers | -| `expiration` | Expiration block of the pre-committed sector | Positive integer | -| `unsealed_cid` | Commitment of data [sector sealing](../glossary.md#commitment-of-data) | Hex string of the unsealed CID bytes | +| Name | Description | Type | +| --------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------- | +| `seal_proof` | Seal proof type this storage provider is using [^note] | String, currently only `StackedDRGWindow2KiBV1P1` is available | +| `sector_number` | The sector number that is being pre-committed | Positive integer | +| `sealed_cid` | [Commitment of replication](../../glossary.md#commitment-of-replication) | Hex string of the sealed CID bytes | +| `deal_ids` | Deal IDs to be pre-committed, from `publish_storage_deals` | Array of integers | +| `expiration` | Expiration block of the pre-committed sector | Positive integer | +| `unsealed_cid` | Commitment of data [sector sealing](../../glossary.md#commitment-of-data) | Hex string of the unsealed CID bytes |
Sectors are not valid after pre-commit. The sectors need to be proven first. @@ -87,9 +87,9 @@ Sectors are not valid after pre-commit. The sectors need to be proven first. [^note]: Only one seal-proof type supported at the moment, `2KiB`. -#### Example +#### Example -Storage provider `//Alice` pre-committing[^pre_commit_sector] a sector number 1, with a single deal ID 0. +Storage provider `//Alice` pre-committing[^pre-committing] a sector number 1, with a single deal ID 0. ```bash storagext-cli --sr25519-key "//Alice" storage-provider pre-commit @pre-commit-sector.json @@ -108,22 +108,23 @@ Where `pre-commit-sector.json` is a file with contents similar to: } ``` -[^pre_commit_sector]: Read more about the `pre-commit` command in [_Storagext CLI/Subcommand `storage-provider`/`pre-commit`_](../storagext-cli/storage-provider.md#pre-commit) +[^pre-committing]: Read more about the `pre-commit` command in [_Storagext CLI/Subcommand `storage-provider`/`pre-commit`_](../../storagext-cli/storage-provider.md#pre-commit) -### `prove_commit_sector` +### `prove_commit_sectors` -After pre-committing some new sectors the storage provider needs to supply a [Proof-of-Replication](../glossary.md#commitment-of-replication) for these sectors [^note]. +After pre-committing some new sectors the storage provider needs to supply a [Proof-of-Replication](../../glossary.md#commitment-of-replication) for these sectors [^note]. +The prove-commit extrinsic takes in an array of the following values: -| Name | Description | Type | -| --------------- | -------------------------------------------------------------------- | ----------------------------- | -| `sector_number` | The sector number that is being prove-committed | Positive integer | -| `proof` | The [proof of replication](../glossary.md#commitment-of-replication) | Hex string of the proof bytes | +| Name | Description | Type | +| --------------- | ----------------------------------------------------------------------- | ----------------------------- | +| `sector_number` | The sector number that is being prove-committed | Positive integer | +| `proof` | The [proof of replication](../../glossary.md#commitment-of-replication) | Hex string of the proof bytes | [^note]: At the moment, any proof of non-zero length is accepted for PoRep. -#### Example +#### Example -This example follows up on the pre-commit example. Storage provider `//Alice` is proven committing[^prove_commit_sector] sector number 1. +This example follows up on the pre-commit example. Storage provider `//Alice` is proven committing[^prove_commit_sectors] sector number 1. ```bash storagext-cli --sr25519-key "//Alice" storage-provider prove-commit @prove-commit-sector.json @@ -138,11 +139,11 @@ Where `prove-commit-sector.json` is a file with contents similar to: } ``` -[^prove_commit_sector]: Read more about the `prove-commit` command in [_Storagext CLI/Subcommand `storage-provider`/`prove-commit`_](../storagext-cli/storage-provider.md#prove-commit) +[^prove_commit_sectors]: Read more about the `prove-commit` command in [_Storagext CLI/Subcommand `storage-provider`/`prove-commit`_](../../storagext-cli/storage-provider.md#prove-commit) ### `submit_windowed_post` -A storage provider needs to periodically submit a [Proof-of-Spacetime](../glossary.md#proofs) to prove that they are still storing the data they promised. Multiple proofs can be submitted at once. +A storage provider needs to periodically submit a [Proof-of-Spacetime](../../glossary.md#proofs) to prove that they are still storing the data they promised. Multiple proofs can be submitted at once. | Name | Description | Type | | ------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------- | @@ -172,7 +173,7 @@ Where `submit-windowed-post.json` is a file with contents similar to: } ``` -[^submit_windowed_post]: Read more about the `submit-windowed-post` command in [_Storagext CLI/Subcommand `storage-provider`/`submit-windowed-post`_](../storagext-cli/storage-provider.md#submit-windowed-post) +[^submit_windowed_post]: Read more about the `submit-windowed-post` command in [_Storagext CLI/Subcommand `storage-provider`/`submit-windowed-post`_](../../storagext-cli/storage-provider.md#submit-windowed-post) ### `declare_faults` @@ -215,7 +216,7 @@ Where `fault-declaration.json` is a file with contents similar to: ] ``` -[^declare_faults]: Read more about the `declare-faults` command in [_Storagext CLI/Subcommand `storage-provider`/`declare-faults`_](../storagext-cli/storage-provider.md#declare-faults) +[^declare_faults]: Read more about the `declare-faults` command in [_Storagext CLI/Subcommand `storage-provider`/`declare-faults`_](../../storagext-cli/storage-provider.md#declare-faults) ### `declare_faults_recovered` @@ -257,7 +258,47 @@ Where `fault-declaration.json` is a file with contents similar to: ] ``` -[^declare_faults_recovered]: Read more about the `declare-faults-recovered` command in [_Storagext CLI/Subcommand `storage-provider`/`declare-faults-recovered`_](../storagext-cli/storage-provider.md#declare-faults-recovered) +[^declare_faults_recovered]: Read more about the `declare-faults-recovered` command in [_Storagext CLI/Subcommand `storage-provider`/`declare-faults-recovered`_](../../storagext-cli/storage-provider.md#declare-faults-recovered) + +### `terminate_sectors` + +A storage provider can terminate sectors with the `terminate_sectors` extrinsic. This requires the storage provider to have no unproven sectors. +`terminate_sectors` can process multiple terminations in a single extrinsic. + +| Name | Description | Type | +| -------------- | --------------------------------------- | ------------------------------------- | +| `terminations` | The sectors and partitions to terminate | An array of termination declarations. | + +Where the termination declarations contain: + +| Name | Description | Type | +| ----------- | -------------------------------------------------------------------------- | ------------------------ | +| `deadline` | The deadline the termination is targeting | Positive integer | +| `partition` | Partition index within the deadline containing the sector to be terminated | Positive integer | +| `sectors` | Sectors in the partition being terminated | Set of positive integers | + +#### Example + +Storage provider `//Alice` terminating sectors[^terminate_sectors] on deadline 0, partition 0, sector 1. + + +```bash +storagext-cli --sr25519-key "//Alice" storage-provider terminate-sectors @terminate-sectors.json +``` + +Where `terminate-sectors.json` is a file with contents similar to: + +```json +[ + { + "deadline": 0, + "partition": 0, + "sectors": [1] + } +] +``` + +[^terminate_sectors]: Read more about the `terminate-sectors` command in [_Storagext CLI/Subcommand `storage-provider`/`terminate-sectors`_](../../storagext-cli/storage-provider.md#terminate-sectors) ## Events @@ -270,12 +311,21 @@ The Storage Provider Pallet emits the following events: - `window_post_proof_type` - The proof type used by the storage provider for sealing sectors. - `sector_size` - Amount of space in each sector committed to the network by the storage provider. - `window_post_partition_sectors` - The number of sectors in each Window PoSt partition (proof). -- `SectorPreCommitted` - A storage provider has pre-committed some new sector after publishing some new deal. +- `SectorsPreCommitted` - A storage provider has pre-committed some new sectors after publishing some new deal. - `owner` - SS58 address of the storage provider. - - `sector` - The sector number being pre-committed. -- `SectorProven` - A storage provider has proven a sector that they previously pre-committed. + - `sectors` - An array with information about the sectors that are pre-committed. This information includes: + - `seal_proof` - The seal proof type used. + - `sector_number` - The sector number that is pre-committed. + - `sealed_cid` - Commitment of replication. + - `deal_ids` - Deal IDs that are activated during pre-commit. + - `expiration` - Expiration of the pre-committed sector. + - `unsealed_cid` - Commitment of data. +- `SectorsProven` - A storage provider has proven a sectors that they previously pre-committed. - `owner` - SS58 address of the storage provider. - - `sector_number` - The sector number that was proven. + - `sectors` - An array with information about the sectors that are proven. This information includes: + - `sector_number` - The sector number that is proven. + - `partition_number` - The partition number the proven sector is in. + - `deadline_idx` - The deadline index assigned to the proven sector. - `SectorSlashed` - A previously pre-committed sector, but not proven, has been slashed by the system because it has expired. - `owner` - SS58 address of the storage provider. - `sector_number` - The sector number that has been slashed because of expiry. @@ -297,6 +347,12 @@ The Storage Provider Pallet emits the following events: - `owner` - SS58 address of the storage provider. - `partition` - Partition number for which the PoSt was missed. - `sectors` - The sectors in the partition declared faulty by the system. +- `SectorsTerminated` - A storage provider has terminated some sectors. + - `owner` - SS58 address of the storage provider. + - `terminations` - An array with information about the terminated sectors. This information includes: + - `deadline` - The deadline to which the terminated sectors were assigned. + - `partition` - The partition number within the deadline containing the terminated sectors. + - `sectors` - The sectors in the partition that have been terminated. ## Errors @@ -315,7 +371,7 @@ The Storage Provider Pallet actions can fail with the following errors: - `SectorNumberAlreadyUsed` - A storage provider tries to pre-commit a sector number that has already been used. - `ExpirationBeforeActivation` - A storage provider tries to pre-commit a sector where that sector expires before activation. - `ExpirationTooSoon` - A storage provider tries to pre-commit a sector with a total lifetime less than MinSectorExpiration. -- `ExpirationTooLong` - A storage provider tries to pre-commit a sector with an expiration that exceeds `MaxSectorExpirationExtension`. +- `ExpirationTooLong` - A storage provider tries to pre-commit a sector with an expiration that exceeds `MaxSectorExpiration`. - `MaxSectorLifetimeExceeded` - A storage provider tries to pre-commit a sector with a total lifetime that exceeds `SectorMaximumLifetime`. - `InvalidCid` - Emitted when a storage provider submits an invalid unsealed CID when trying to pre-commit a sector. - `ProveCommitAfterDeadline` - A storage provider has tried to prove a previously pre-committed sector after the proving deadline. @@ -323,31 +379,54 @@ The Storage Provider Pallet actions can fail with the following errors: - `InvalidUnsealedCidForSector` - This error is emitted when the declared unsealed_cid for pre_commit is different from the one calculated by the system. - `FaultDeclarationTooLate` - A fault declaration was submitted after the fault declaration cutoff. The fault declaration can be submitted after the upcoming deadline is closed. - `FaultRecoveryTooLate` - A fault recovery was submitted after the fault recovery cutoff. The fault recovery can be submitted after the upcoming deadline is closed. -- `DeadlineError` - An error was encountered in the deadline module. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. -- `PartitionError` - An error was encountered in the partition module. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. -- `StorageProviderError` - An error was encountered in the storage provider module. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. -- `SectorMapError` - An error was encountered in the sector map module. These errors can be: - - `FailedToInsertSector` - Internal bounds violation with Sectors. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. - - `FailedToInsertPartition` - Internal bounds violation with partitions. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. -- `CouldNotActivateSector` - Failure during prove commit when trying to convert a previously pre-committed sector due to a programming error. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. -- `CouldNotVerifySectorForPreCommit` - Failure during pre-commit due to the [commd](../glossary.md#commitment-of-data) calculation failing due to a programming error. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. +- `CouldNotTerminateDeals` - Emitted when trying to terminate sector deals fails. +- `InvalidDeadlineSubmission` - Emitted when an error occurs when submitting PoSt. +- `CouldNotVerifySectorForPreCommit` - Failure during pre-commit due to the [commd](../../glossary.md#commitment-of-data) calculation failing due to a programming error. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. - `SlashingFailed` - Slashing of funds fails due to a programmer error. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. - `ConversionError` - Due to a programmer error. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. +- `GeneralPalletError` - An error ocurred in on of the pallet modules. These errors can be: + - `PartitionErrorFailedToAddSector` - Emitted when adding sectors fails. + - `PartitionErrorDuplicateSectorNumber` - Emitted when trying to add a sector number that has already been used in this partition. + - `PartitionErrorFailedToAddFaults` - Emitted when adding in the partition faults fails. + - `PartitionErrorSectorsNotLive` - Emitted when trying to remove sectors that are not live. + - `PartitionErrorFailedToRemoveRecoveries` - Emitted when removing recovering sectors from the partition fails. + - `PartitionErrorUnexpectedRecoveries` - Emitted when encountering unexpected recoveries while popping expired sectors. + - `PartitionErrorExpiredSectorsAlreadyTerminated` - Emitted when trying to pop expired sectors that are already terminated. + - `DeadlineErrorDeadlineIndexOutOfRange` - Emitted when the passed in deadline index supplied for `submit_windowed_post` is out of range. + - `DeadlineErrorDeadlineNotFound` - Emitted when a trying to get a deadline index but fails because that index does not exist. + - `DeadlineErrorCouldNotConstructDeadlineInfo` - Emitted when constructing `DeadlineInfo` fails. + - `DeadlineErrorPartitionAlreadyProven` - Emitted when a proof is submitted for a partition that is already proven. + - `DeadlineErrorPartitionNotFound` - Emitted when trying to retrieve a partition that does not exit. + - `DeadlineErrorProofUpdateFailed` - Emitted when trying to update proven partitions fails. + - `DeadlineErrorMaxPartitionsReached` - Emitted when max partition for a given deadline have been reached. + - `DeadlineErrorCouldNotAddSectors` - Emitted when trying to add sectors to a deadline fails. + - `DeadlineErrorSectorsNotFound` - Emitted when trying to use sectors which haven't been prove committed yet. + - `DeadlineErrorSectorsNotFaulty` - Emitted when trying to recover non-faulty sectors. + - `DeadlineErrorCouldNotAssignSectorsToDeadlines` - Emitted when assigning sectors to deadlines fails. + - `DeadlineErrorFailedToUpdateFaultExpiration` - Emitted when trying to update fault expirations fails. + - `StorageProviderErrorMaxPreCommittedSectorExceeded` - Happens when an SP tries to pre-commit more sectors than SECTOR_MAX. + - `StorageProviderErrorSectorNotFound` - Happens when trying to access a sector that does not exist. + - `StorageProviderErrorSectorNumberInUse` - Happens when a sector number is already in use. + - `SectorMapErrorFailedToInsertSector` - Emitted when trying to insert sector(s) fails. + - `SectorMapErrorFailedToInsertPartition` - Emitted when trying to insert partition fails. + - `ExpirationQueueErrorExpirationSetNotFound` - Expiration set not found. + - `ExpirationQueueErrorSectorNotFound` - Sector not found in expiration set. + - `ExpirationQueueErrorInsertionFailed` - Insertion into the expiration queue failed. ## Pallet constants The Storage Provider Pallet has the following constants: -| Name | Description | Value | -| ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | -| `WPoStProvingPeriod` | The average [period](../glossary.md#proving-period) for proving all sectors maintained by a storage provider. | 4 Minutes | -| `WPoStChallengeWindow` | The period immediately before a deadline during which a challenge can be generated by the chain and the requisite proofs computed. | 2 Minutes | -| `WPoStChallengeLookBack` | This period allows the storage providers to start working on the [PoSt](../glossary.md#post) before the deadline is officially opened to receiving a [PoSt](../glossary.md#post). | 1 Minute | -| `WPoStPeriodDeadlines` | Represents how many challenge deadlines there are in one proving period. Closely tied to `WPoStChallengeWindow`. | 48 | -| `MinSectorExpiration` | Minimum time past the current block a sector may be set to expire. | 5 Minutes | -| `MaxSectorExpirationExtension` | Maximum time past the current block a sector may be set to expire. | 60 Minutes | -| `SectorMaximumLifetime` | Maximum time a sector can stay in pre-committed state. | 120 Minutes | -| `MaxProveCommitDuration` | Maximum time between [pre-commit](#pre_commit_sector) and [proving](#prove_commit_sector) the committed sector. | 5 Minutes | -| `MaxPartitionsPerDeadline` | Maximum number of partitions that can be assigned to a single deadline. | 3000 | -| `FaultMaxAge` | Maximum time a [fault](../glossary.md#fault) can exist before being removed by the pallet. | 210 Minutes | -| FaultDeclarationCutoff | Time before a deadline opens that a storage provider can declare or recover a fault. | 2 Minutes | +| Name | Description | Value | +| ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- | +| `WPoStProvingPeriod` | The average [period](../../glossary.md#proving-period) for proving all sectors maintained by a storage provider. | 4 Minutes (40 Blocks) | +| `WPoStChallengeWindow` | The period immediately before a deadline during which a challenge can be generated by the chain and the requisite proofs computed. | 2 Minutes (20 Blocks) | +| `WPoStChallengeLookBack` | This period allows the storage providers to start working on the [PoSt](../../glossary.md#post) before the deadline is officially opened to receiving a [PoSt](../../glossary.md#post). | 1 Minute (10 Blocks) | +| `WPoStPeriodDeadlines` | Represents how many challenge deadlines there are in one proving period. Closely tied to `WPoStChallengeWindow`. | 48 | +| `MinSectorExpiration` | Minimum time past the current block a sector may be set to expire. | 5 Minutes (50 Blocks) | +| `MaxSectorExpiration` | Maximum time past the current block a sector may be set to expire. | 60 Minutes (600 Blocks) | +| `SectorMaximumLifetime` | Maximum time a sector can stay in pre-committed state. | 120 Minutes (1200 Blocks) | +| `MaxProveCommitDuration` | Maximum time between [pre-commit](#pre_commit_sectors) and [proving](#prove_commit_sectors) the committed sector. | 5 Minutes (50 Blocks) | +| `MaxPartitionsPerDeadline` | Maximum number of partitions that can be assigned to a single deadline. | 3000 | +| `FaultMaxAge` | Maximum time a [fault](../../glossary.md#fault) can exist before being removed by the pallet. | 210 Minutes (2100 Blocks) | +| FaultDeclarationCutoff | Time before a deadline opens that a storage provider can declare or recover a fault. | 2 Minutes (20 Blocks) | diff --git a/src/architecture/polka-storage-provider-server.md b/src/architecture/polka-storage-provider-server.md new file mode 100644 index 0000000..84746a3 --- /dev/null +++ b/src/architecture/polka-storage-provider-server.md @@ -0,0 +1,119 @@ +# Polka Storage Provider — Server Architecture + + +The server has two main fronts, the JSON-RPC API which provides an interface for users to submit deal proposals to the storage provider, +and the HTTP API which consists of a single endpoint where users are to submit their data — as illustrated below. + +

+ +The user is first required to propose a deal, which once accepted by the storage provider (signaled by the return of a CID) allows the user to submit a file +(using `curl` for example) to the server; finally the user can then publish a signed deal to the storage provider. For more details, see the [File Upload Demo](../getting-started/demo-file-store.md). + +The responsibility then falls on the storage provider to seal, publish and activate the deal on the Polka Storage parachain. + +## JSON-RPC API + +The JSON-RPC endpoint exposes the following methods: + +
+info — which returns information about the Storage Provider. +

JSON-RPC Example

+
+{
+  "jsonrpc": "2.0",
+  "id": 0,
+  "method": "v0_info",
+  "params": []
+}
+
+
+ +
+propose_deal — accepts a deal proposal and returns a CID for the file upload. +

JSON-RPC Example

+
+{
+  "jsonrpc": "2.0",
+  "id": 0,
+  "method": "v0_propose_deal",
+  "params": [
+    {
+      "piece_cid": "baga6ea4seaqj527iqfb2kqhy3tmpydzroiigyaie6g3txai2kc3ooyl7kgpeipi",
+      "piece_size": 2048,
+      "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
+      "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y",
+      "label": "",
+      "start_block": 200,
+      "end_block": 250,
+      "storage_price_per_block": 500,
+      "provider_collateral": 1250,
+      "state": "Published"
+    }
+  ]
+}
+
+
+ +
+publish_deal — after a file has been uploaded, accepts a signed deal for publishing. +
+{
+  "jsonrpc": "2.0",
+  "id": 0,
+  "method": "v0_publish_deal",
+  "params": [
+    {
+      "deal_proposal": {
+        "piece_cid": "baga6ea4seaqj527iqfb2kqhy3tmpydzroiigyaie6g3txai2kc3ooyl7kgpeipi",
+        "piece_size": 2048,
+        "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
+        "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y",
+        "label": "",
+        "start_block": 100000,
+        "end_block": 100050,
+        "storage_price_per_block": 500,
+        "provider_collateral": 1250,
+        "state": "Published"
+      },
+      "client_signature": {
+        "Sr25519": "c835a1c5215fc017067d30a8f49df0c643233881e57d8bd7232f695e1d28c748e8872b45712dcb403e28792cd1fb2b6161053b3344d4f6664bafec77349abd80"
+      }
+    }
+  ]
+}
+
+
+ +## HTTP API + +The HTTP API exposes a single PUT method — `/upload/` where `` is the CID returned as a result of `propose_deal`. + + +## Sealing Pipeline + +As shown in the previous illustration, the sealing pipeline is responsible for gathering pieces into sectors, sealing said sectors and proving their storage. +To achieve that, the pipeline is (currently) composed of 3 main stages. + +![The sealine pipeline, currently composed of the stages: Add Piece, Pre Commit and Prove Commit](../images/architecture/storage-provider-pipeline.png) + +### Add Piece + +The Add Piece stage gathers pieces into unsealed sectors, preparing them for the next steps. + +Given we're currently only supporting sectors with 2KiB size, we're converting single pieces into sectors — +when a piece comes in, we convert it to a single sector, without gathering multiple pieces. + +### Pre Commit + +By itself, the Pre Commit has two inner stages — Pre Commit 1 (PC1) and Pre Commit 2 (PC2). +PC1 is responsible for generating the [Stacked Depth Robust Graph](https://spec.filecoin.io/algorithms/sdr/#section-algorithms.sdr.stacked-depth-robust-graphs), +while PC2 is responsible for handling the construction of the [Merkle tree and proofs](https://spec.filecoin.io/algorithms/sdr/#section-algorithms.sdr.merkle-proofs). +After this process is completed, the Pre Commit information is submitted to the chain for verification. + +### Prove Commit + +The Prove Commit stage is where the [Proof of Replication](https://docs.filecoin.io/basics/the-blockchain/proofs#proof-of-replication-porep) is generated, +after generation it is submitted to the network for validation and the sector is finally marked as `Active`, +signaling that the Storage Provider has effectively stored the sector and is ready to start performing regular proof submissions. + + diff --git a/src/getting-started/building/docker.md b/src/getting-started/building/docker.md new file mode 100644 index 0000000..b3e750e --- /dev/null +++ b/src/getting-started/building/docker.md @@ -0,0 +1,78 @@ +# Docker Setup + +This guide will outline how to setup your environment using Docker to get started with the Polka Storage parachain. + +## Pre-requisites + +Install Docker on your system by following the [Docker install instructions](https://docs.docker.com/engine/install/). + +> Using Podman instead of Docker **may** work, however, we **do not support** Podman! + +## Dockerfile setup + +All docker builds are composed of 4 stages. + +1. Set up [`cargo chef`](https://github.com/LukeMathWalker/cargo-chef), this caches the Rust dependencies for faster builds. +2. Planning — `cargo chef` analyzes the current project to determine the minimum subset of file required to build it an cache the dependencies. +3. Build — `cargo chef` checks the project skeleton identified in the planner stage and builds it to cache dependencies. +4. Runtime — sets up the runtime with Debian and imports the binary build in the previous stage. + +## Building & Running + +Clone the repository and go into the directory: + +```shell +git clone git@github.com:eigerco/polka-storage.git +cd polka-storage +``` + +You can find Dockerfiles for each binary under the `docker/` folder. +To build the images manually you can use the following command: + +```bash +docker build \ + --build-arg VCS_REF="$(git rev-parse HEAD)" \ + --build-arg BUILD_DATE="$(date -u +'%Y-%m-%dT%H:%M:%SZ')" \ + -t :"$(cargo metadata --format-version=1 --no-deps | jq -r '.packages[0].version')" \ + --file ./docker/dockerfiles/.Dockerfile \ + . +``` + +Where you can replace `` by one of the following: + +- `polka-storage-node` +- `polka-storage-provider-server` +- `polka-storage-provider-client` +- `mater-cli` +- `storagext-cli` + +To run the images manually, you apply the same pattern to the following command: + +```bash +docker run -it polkadotstorage.azurecr.io/:"$(cargo metadata --format-version=1 --no-deps | jq -r '.packages[0].version')" +``` + +### Just recipes + +To simplify the building process, we've written some [Just](https://github.com/casey/just) recipes. + +#### Build recipes + +| Command | Description | +| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | +| `build-mater-docker` | Builds the `mater` CLI image which is used by storage clients to convert files to CARv2 format and extract CARv2 content. | +| `build-polka-storage-node-docker` | Builds the Polka Storage parachain node image. | +| `build-polka-storage-provider-server-docker` | Builds the Storage Provider server image. | +| `build-polka-storage-provider-client-docker` | Builds the Storage Provider client image. | +| `build-storagext-docker` | Builds the `storagext` CLI image used to execute extrinsics. | +| `build-docker-all` | Builds all the images above, this might take a while to complete. | + +#### Running recipes + +| Command | Description | +| ------------------------------------------ | ------------------------------------------------------------------------------------------ | +| `run-mater-docker` | Runs the image, opening a shell with access to the `mater-cli` binary. | +| `run-polka-storage-node-docker` | Runs the `polka-storage-node` inside the built Docker image. | +| `run-polka-storage-provider-server-docker` | Runs the image, opening a shell with access to the `polka-storage-provider-server` binary. | +| `run-polka-storage-provider-client-docker` | Runs the image, opening a shell with access to the `polka-storage-provider-client` binary. | +| `run-storagext-docker` | Runs the image, opening a shell with access to the `storagext-cli` binary. | diff --git a/src/getting-started/building/index.md b/src/getting-started/building/index.md new file mode 100644 index 0000000..0313301 --- /dev/null +++ b/src/getting-started/building/index.md @@ -0,0 +1,26 @@ +# Building + +The following chapters will cover how to build the Polka Storage parachain using multiple methods. + +
+Quick reminder that Windows is not part of the supported operating systems. +As such, the following guides have not been tested on Windows. +
+ + +We'll be building 5 artifacts: + +* `polka-storage-node` — the Polka Storage Polkadot parachain node. +* `polka-storage-provider-server` — the Polka Storage Storage Provider, + responsible for accepting deals, storing files and executing storage proofs. +* `polka-storage-provider-client` — the Polka Storage Storage Provider client, + this CLI tool has several utilities to interact with the Storage Provider server, + such as proposing and publishing deals, as well as some wallet utilities and proof demos. +* `mater-cli` — the Mater CLI enables you to convert files into CARv2 archives, + an essential part of preparing files for submission to the network. +* `storagext-cli` — the Storagext CLI is a lower-level tool to manually run the Polka Storage extrinsics. + +To build these artifacts, we provide two main methods: + +* [Building from source](./source.md) +* [Building with Docker](./docker.md) diff --git a/src/getting-started/building/source.md b/src/getting-started/building/source.md new file mode 100644 index 0000000..59126ed --- /dev/null +++ b/src/getting-started/building/source.md @@ -0,0 +1,129 @@ +# Building from source + +This guide will outline how to setup your environment to build the Polka Storage parachain, +we cover how to build the binaries directly on your system or using [Nix](https://nixos.org/download/) to ease the process. + +* [Get the code](#get-the-code) +* [System dependencies](#system-dependencies) +* [Using Nix](#using-nix) + * [Pre-requisites](#pre-requisites) +* [Building](#building) + +## Get the code + +To get started, first clone the repository and enter the repository's directory: + +```bash +git clone git@github.com:eigerco/polka-storage.git +cd polka-storage +``` + + +## System dependencies + +To build the binaries directly on your system you will need the following tools: + +* Rust 1.77 — you can install it using [`rustup`](https://rustup.rs/) and its [guide](https://rust-lang.github.io/rustup/installation/other.html) for help. +* Other dependencies — keep reading, we'll get to it after the end of this list! +* `just` (optional) — (after installing Rust) you can use `cargo install just` or check the [official list of packages](https://just.systems/man/en/packages.html). + +The dependencies mentioned are for Linux distros using the `apt` family of package managers. +Different systems may use different package managers, as such, they may require you to find the equivalent package. + +To install the required dependencies run the following commands: + +```shell +$ sudo apt update +$ sudo apt install -y libhwloc-dev \ + opencl-headers \ + ocl-icd-opencl-dev \ + protobuf-compiler \ + clang \ + build-essential \ + git \ + curl +``` + +## Using Nix + +You can use Nix to simplify the building process, +if you're just taking the network for test-drive this is a great method to get started. + +Nix will take care of setting up all the dependencies for you! +If you're curious, you can read more about using Nix in [fasterthanlime's blog](https://fasterthanli.me/series/building-a-rust-service-with-nix/part-9), +the [official Nix guide](https://nixos.org/learn/) or [Determinate Systems' Zero to Nix guide](https://zero-to-nix.com/). + +
+Binaries built using Nix will not work on other systems since they will be linked with Nix specific paths. +
+ +### Pre-requisites + +- `nix` — which you can install by following the [official guide](https://nixos.org/download/) + or using the [Determinate Systems installer](https://github.com/DeterminateSystems/nix-installer) — the latter being usually more reliable on MacOS systems. +- `direnv` (optional) — which you can install by following the [official guide](https://direnv.net/docs/installation.html) + +If you're using `direnv`, when going into the cloned directory for the first time `nix` will activate automatically and +install the required packages, this make take some time. + +If you're _not_ using `direnv`, you will need to run `nix develop` to achieve the same effect — +for more information refer to the official Nix guide — https://nix.dev/manual/nix/2.17/command-ref/new-cli/nix3-develop. + +## Building + +After all this setup, it is time to start building the binaries, which you can do manually using the following command: + +
+ +When building `polka-storage-node` you should add `--features polka-storage-runtime/testnet` which enables the testnet configuration; all the code in the repo is currently targeting this feature, not including it may lead to unexpected behavior. + +When building `storagext-cli` you may want to add `--features storagext/insecure_url` which enables using non-TLS HTTP and WebSockets. +
+ +```bash +cargo build --release -p +``` + +Where `` is one of: + +- `polka-storage-node` +- `polka-storage-provider-client` +- `polka-storage-provider-server` +- `storagext-cli` +- `mater-cli` + + +For more information on what each binary does, refer to [Building](./index.md). + +### Just recipes + +To simplify the building process, we've written some [Just](https://github.com/casey/just) recipes. + +| Command | Description | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | +| `build-polka-storage-node` | Builds the Polka Storage parachain node. | +| `build-polka-storage-provider-server` | Builds the Storage Provider server binary. | +| `build-polka-storage-provider-client` | Builds the Storage Provider client binary. | +| `build-storagext-cli` | Builds the `storagext` CLI used to execute extrinsics. | +| `build-mater-cli` | Builds the `mater` CLI which is used by storage clients to convert files to CARv2 format and extract CARv2 content. | +| `build-binaries-all` | Builds all the binaries above, this may take a while (but at least `cargo` reuses artifacts). | + +## Running + +After building the desired binaries, you can find them under the `target/release` folder +(or `target/debug` if you didn't use the `-r`/`--release` flag). + +Assuming you're in the project root, you can run them with the following command: + +```bash +$ target/release/ +``` + +Where `` is one of: +* `polka-storage-node` +* `polka-storage-provider-server` +* `polka-storage-provider-client` +* `mater-cli` +* `storagext-cli` + +> Additionally, you can move them to a folder under your `$PATH` and run them as you would with any other binary. diff --git a/src/getting-started/car-server.md b/src/getting-started/car-server.md deleted file mode 100644 index 4281e0a..0000000 --- a/src/getting-started/car-server.md +++ /dev/null @@ -1,60 +0,0 @@ -# CAR server - -It is an HTTP server that converts arbitrary content into a [CARv2](https://ipld.io/specs/transport/car/carv2/) file and serves it over HTTP - supporting the latest CARv2 format, -which is not yet entirely supported by other crates in the Rust ecosystem. -The next steps describe how to run the server locally and use it to upload and download files. - -
-The server is a proof of concept, showcasing our CARv2 implementation, but it is not intended to be used in production. -Anyone can upload and download files without authentication or authorization. -
- -## Start the server - -1. Create a Docker volume to store uploaded files: - -`docker volume create storage_provider` - -2. Start the server: - -``` -docker run \ - -p 127.0.0.1:9000:9000 \ - --mount source=storage_provider,destination=/app/uploads \ - polkadotstorage.azurecr.io/polka-storage-provider:0.1.0 storage \ - --listen-addr 0.0.0.0:9000 -``` - -- `-p 127.0.0.1:9000:9000`: Maps port `9000` on the localhost to port `9000` on the container. -- `--mount source=storage_provider,destination=/app/uploads`: Mounts the `storage_provider` volume to `/app/uploads` inside the container. -- `polkadotstorage.azurecr.io/polka-storage-provider:0.1.0 storage`: Runs the `polkadotstorage.azurecr.io/polka-storage-provider:0.1.0` image with the `storage` command. -- `--listen-addr 0.0.0.0:9000`: Configures the server to listen on all available network interfaces. - -## Verifying the Setup - -After setting up and starting the CAR server, it's essential to verify that everything works correctly. -Follow these steps to ensure your setup works as expected: - -1. Upload a test file using the instructions in the [Upload a file](../storage-provider-cli/storage.md#upload-a-file) section. Make sure to note the CID returned by the server. - -2. Download the CAR file using the retrieved CID, following the steps in the [Download the CAR File](../storage-provider-cli/storage.md#download-the-car-file) section. - -3. [Optional] Verify the contents of the downloaded CAR file. Using, for example, [go-car](https://github.com/ipld/go-car/tree/master/cmd/car#install)'s `inspect` command: - - ```bash - car inspect - ``` - - The user can use `debug` for more detailed output: - - ```bash - car debug - ``` - - If the user desires, they can extract the contents of the file: - - ```bash - car extract - ``` - -If a file can be successfully uploaded, the server produces a CID, allows download the corresponding CAR file, and verify its contents, the CAR server setup is working correctly. diff --git a/src/getting-started/demo-file-prove.md b/src/getting-started/demo-file-prove.md new file mode 100644 index 0000000..900587c --- /dev/null +++ b/src/getting-started/demo-file-prove.md @@ -0,0 +1,56 @@ +# Proving a file + +To store the file according to the protocol, Storage Provider has to assign it to a sector, pre-commit and then prove it! +That's a lot of steps, but this is handled automatically, behind the scenes by the [pipeline](../architecture/polka-storage-provider-server.md#sealing-pipeline), then the file is eventually [published](../storage-provider-cli/client/index.md#publish-deal). + +Here are excerpts from Storage Provider Node after executing the [store a file scenario](./demo-file-store.md): + +```log +2024-11-11T12:34:21.430693Z INFO start_rpc_server: polka_storage_provider_server::rpc: Starting RPC server at 127.0.0.1:8000 +2024-11-11T12:34:21.430870Z INFO start_upload_server: polka_storage_provider_server::storage: Starting HTTP storage server at: 127.0.0.1:8001 +2024-11-11T12:34:21.431984Z INFO start_rpc_server: polka_storage_provider_server::rpc: RPC server started +2024-11-11T12:35:07.883255Z INFO request{method=PUT matched_path="/upload/:cid" request_id=e71d7e49-0272-435e-899e-a12a5d639268}:upload: polka_storage_provider_server::storage: CAR file created final_content_path="/var/folders/51/ch08ltd95bxcwpvskd28wr5h0000gp/T/Xvm5m7j/deals_storage/car/bafkreihoxd7eg2domoh2fxqae35t7ihbonyzcdzh5baevxzrzkaakevuvy.car" +2024-11-11T12:37:29.258216Z INFO add_piece: polka_storage_provider_server::pipeline: Adding a piece... +2024-11-11T12:37:29.258785Z INFO polka_storage_provider_server::pipeline: Preparing piece... +2024-11-11T12:37:29.259375Z INFO polka_storage_provider_server::pipeline: Adding piece... +2024-11-11T12:37:29.261621Z INFO add_piece: polka_storage_provider_server::pipeline: Finished adding a piece +2024-11-11T12:37:29.261979Z INFO polka_storage_provider_server::pipeline: Add Piece for piece Commitment { commitment: [...], kind: Piece }, deal id 0, finished successfully. +2024-11-11T12:37:29.262023Z INFO precommit: polka_storage_provider_server::pipeline: Starting pre-commit +2024-11-11T12:37:29.262258Z INFO precommit: polka_storage_provider_server::pipeline: Padded sector, commencing pre-commit and getting last finalized block +2024-11-11T12:37:29.263185Z INFO precommit: polka_storage_provider_server::pipeline: Current block: 35 +2024-11-11T12:37:29.263852Z INFO filecoin_proofs::api::seal: seal_pre_commit_phase1:start: SectorId(1) +2024-11-11T12:37:29.275251Z INFO storage_proofs_porep::stacked::vanilla::proof: replicate_phase1 +2024-11-11T12:37:29.275814Z INFO storage_proofs_porep::stacked::vanilla::graph: using parent_cache[64 / 64] +2024-11-11T12:37:29.276105Z INFO storage_proofs_porep::stacked::vanilla::cache: parent cache: opening /var/tmp/filecoin-parents/v28-sdr-parent-3f0eef38bb48af1f48ad65e14eb85b4ebfc167cec18cd81764f6d998836c9899.cache, verify enabled: false +2024-11-11T12:37:29.277633Z INFO storage_proofs_porep::stacked::vanilla::proof: single core replication +2024-11-11T12:37:29.277644Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: generate labels +2024-11-11T12:37:29.277681Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: generating layer: 1 +2024-11-11T12:37:29.277915Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: storing labels on disk +2024-11-11T12:37:29.278316Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: generated layer 1 store with id layer-1 +2024-11-11T12:37:29.278328Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: setting exp parents +2024-11-11T12:37:29.278336Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: generating layer: 2 +2024-11-11T12:37:29.278418Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: storing labels on disk +2024-11-11T12:37:29.278735Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: generated layer 2 store with id layer-2 +2024-11-11T12:37:29.278745Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: setting exp parents +2024-11-11T12:37:29.278761Z INFO filecoin_proofs::api::seal: seal_pre_commit_phase1:finish: SectorId(1) +[...] +2024-11-11T12:37:29.313831Z INFO storage_proofs_core::data: dropping data /var/folders/51/ch08ltd95bxcwpvskd28wr5h0000gp/T/Xvm5m7j/deals_storage/sealed/1 +2024-11-11T12:37:29.314137Z INFO filecoin_proofs::api::seal: seal_pre_commit_phase2:finish +2024-11-11T12:37:29.314165Z INFO precommit: polka_storage_provider_server::pipeline: Created sector's replica: PreCommitOutput { } +[...] +2024-11-11T12:37:57.324204Z INFO precommit: polka_storage_provider_server::pipeline: Successfully pre-commited sectors on-chain: [SectorsPreCommitted { block: 39, [...] }] +2024-11-11T12:37:57.324292Z INFO polka_storage_provider_server::pipeline: Precommit for sector 1 finished successfully. +2024-11-11T12:37:57.324345Z INFO prove_commit: polka_storage_provider_server::pipeline: Starting prove commit +2024-11-11T12:37:57.325705Z INFO prove_commit: polka_storage_provider_server::pipeline: Wait for block 49 to get randomness +2024-11-11T12:39:05.518784Z INFO storage_proofs_porep::stacked::vanilla::proof: generating interactive vanilla proofs +2024-11-11T12:39:05.529259Z INFO bellperson::groth16::prover::native: Bellperson 0.26.0 is being used! +2024-11-11T12:39:06.634632Z INFO bellperson::groth16::prover::native: synthesis time: 1.105318708s +2024-11-11T12:39:06.634659Z INFO bellperson::groth16::prover::native: starting proof timer +[...] +2024-11-11T12:39:23.728566Z INFO bellperson::groth16::prover::native: prover time: 17.094277959s +2024-11-11T12:39:23.737186Z INFO prove_commit: polka_storage_provider_server::pipeline: Proven sector: 1 +``` + +After that, Storage Provider needs to continously [submit a PoSt](../architecture/pallets/storage-provider.md#submit_windowed_post) to prove that they are still storing the file. If they do not, they'll be slashed. +We have not yet integrated the logic for PoSt verification with Storage Provider node, but the logic on-chain has been implemented. + diff --git a/src/getting-started/demo-file-store.md b/src/getting-started/demo-file-store.md new file mode 100644 index 0000000..74871c6 --- /dev/null +++ b/src/getting-started/demo-file-store.md @@ -0,0 +1,149 @@ +# Storing a file + +
+Before reading this guide, please follow the local testnet guide and storage provider guide. +You should have a working testnet and a Storage Provider running! +
+ + +## Storage Client +The Polkadot logo + +Alice is a [Storage User](../glossary.md#storage-user) and wants to store an image of her lovely Polkadot logo [`polkadot.svg`](../images/polkadot.svg) in the Polka Storage [parachain](../glossary.md#parachain). + +Alice knows that she needs to prepare an image for storage and get its [CID](https://github.com/multiformats/cid). +To do so, she first converts it into a [CARv2 archive](https://ipld.io/specs/transport/car/carv2/) and gets the piece cid. + +```bash +$ mater-cli convert -q --overwrite polkadot.svg polkadot.car +bafkreihoxd7eg2domoh2fxqae35t7ihbonyzcdzh5baevxzrzkaakevuvy +$ polka-storage-provider-client proofs commp polkadot.car +{ + "cid": "baga6ea4seaqabpfwrqjcwrb4pxmo2d3dyrgj24kt4vqqqcbjoph4flpj2e5lyoq", + "size": 2048 +} +``` + +### Proposing a deal + +Afterwards, it's time to propose a deal, currently — i.e. while the network isn't live — +any deals will be accepted by Charlie (the Storage Provider). + + +Alice fills out the deal form according to a JSON template (`polka-logo-deal.json`): + +```json +{ + "piece_cid": "baga6ea4seaqabpfwrqjcwrb4pxmo2d3dyrgj24kt4vqqqcbjoph4flpj2e5lyoq", + "piece_size": 2048, + "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", + "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", + "label": "", + "start_block": 200, + "end_block": 250, + "storage_price_per_block": 500, + "provider_collateral": 1000100200, + "state": "Published" +} +``` + +* `piece_cid` — is the `cid` field from the previous step, where she calculated the piece commitment. It uniquely identifies the piece. +* `piece_size` — is the `size` field from the previous step, where she calculated the piece commitment. It is the size of the processed piece, not the original file! +* `client` — is the client's (i.e. the reader's) public key, encoded in bs58 format. + For more information on how to generate your own keypair, read the [Polka Storage Provider CLI/`client`/`wallet`](../storage-provider-cli/client/wallet.md). +* `provider` — is the storage provider's public key, encoded in bs58 format. + If you don't know your storage provider's public key, you can query it using `polka-storage-provider-client`'s `info` command. +* `label` — is an arbitrary string to be associated with the deal. +* `start_block` — is the deal's start block, it MUST be positive and lower than `end_block`. +* `end_block` — is the deal's end block, it must be positive and larger than `start_block`. +* `storage_price_per_block` — the storage price over the duration of a single block — e.g. if your deal is 20 blocks long, it will cost `20 * storage_price_per_block` in total. +* `provider_collateral` — the price to pay *by the storage provider* if they fail to uphold the deal. +* `state` — the deal state, only `Published` is accepted. + + +
+ +The `start_block` and `end_block` fields may need to be changed depending on the current block you are on. +The values `200` and `250` are solely for demonstration purposes and we encourage you to try other values! + +
+ +
+Variables subject to change depending on the chains state + +`start_block` - The start block **must** be after the current block. Check the polka storage node logs or use the polkadot.js UI for the current block and adjust the start_block value accordingly. + +`end_block` - The end block **must** be between 50 and 1800 blocks after `start_block`. + +See the [Storage Provider Constants](../architecture/pallets/storage-provider.md#pallet-constants) and the [Market Constant](../architecture/pallets/market.md#constants) for more information about the configuration variables + +
+ +When the deal is ready, she proposes it: + +```bash +$ polka-storage-provider-client propose-deal --rpc-server-url "http://localhost:8000" "@polka-logo-deal.json" +bagaaierab543mpropvi5mnmtptytnnlbr2j7vea7lowcugrqt7epanybw7ta +``` + +The storage provider replied with a CID — the CID of the deal Alice just sent — she needs to keep this CID for the next steps! + +Once the server has replied with the CID, she's ready to upload the file. +This can be done with just any tool that can upload a file over HTTP. +The server supports both [multipart forms](https://curl.se/docs/httpscripting.html#file-upload-post) and [`PUT`](https://curl.se/docs/httpscripting.html#put). + +```bash +$ curl --upload-file "polkadot.svg" "http://localhost:8001/upload/bagaaierab543mpropvi5mnmtptytnnlbr2j7vea7lowcugrqt7epanybw7ta" +baga6ea4seaqabpfwrqjcwrb4pxmo2d3dyrgj24kt4vqqqcbjoph4flpj2e5lyoq +``` + +### Publishing the deal + +Before Alice publishes a deal, she must ensure that she has the necessary funds available in the market escrow, to be able to pay for the deal: + +```bash +$ storagext-cli --sr25519-key "//Alice" market add-balance 25000000000 +[0x6489…a2c0] Balance Added: { account: 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY, amount: 25000000000 } +``` + +Finally, she can publish the deal by submitting her deal proposal along with your signature to the storage provider. + +To sign her deal proposal she runs: + +```bash +$ polka-storage-provider-client sign-deal --sr25519-key "//Alice" @polka-logo-deal.json +``` + +```json +{ + "deal_proposal": { + "piece_cid": "baga6ea4seaqabpfwrqjcwrb4pxmo2d3dyrgj24kt4vqqqcbjoph4flpj2e5lyoq", + "piece_size": 2048, + "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", + "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", + "label": "", + "start_block": 200, + "end_block": 250, + "storage_price_per_block": 500, + "provider_collateral": 1000100200, + "state": "Published" + }, + "client_signature": { + "Sr25519": "7eb8597441711984b7352bd4a118eac57341296724c20d98a76ff8d01ee64038f6a9881e492a98c3a190e7b600a8313d72e9f0edacb3e6df6b0b4507dabb9580" + } +} +``` + +> Hint: you can write the following command to *just* get the file: +> ``` +> polka-storage-provider-client sign-deal --sr25519-key "//Alice" @polka-logo-deal.json > signed-logo-deal.json +> ``` + +All that's left is to [publish the deal](../storage-provider-cli/client/index.md#publish-deal): + +```bash +$ polka-storage-provider-client publish-deal --rpc-server-url "http://localhost:8000" @signed-logo-deal.json +Successfully published deal of id: 0 +``` + +On Alice's side, that's it! diff --git a/src/getting-started/demo.md b/src/getting-started/demo.md deleted file mode 100644 index 91cfaf0..0000000 --- a/src/getting-started/demo.md +++ /dev/null @@ -1,263 +0,0 @@ -# Real-world use case demo - -
-Before reading this guide, please follow the local testnet guide and have a working testnet running! -
- -
- -For convenience's sake, we have a script that automates the actions in this guide. -The script is available at the following link: - - -Alternatively, download and run in a single step: - -```bash -wget https://polka-storage.s3.eu-central-1.amazonaws.com/demo.sh -chmod +x demo.sh -./demo.sh -``` - -
- -The script has some pre-requisites: -* The storagext-cli binary *must* exist and be present in the $PATH — - instructions on how to achieve that are available in the Local Testnet - Polka Storage Parachain chapter. -* You need to launch a *fresh* parachain, as the script needs to be run while the Charlie node is booting up; - the script will wait for the first block before starting. *This process may need more than one attempt.* - -
- -
- -A high-level overview with diagrams of the process described below can be found in Pallets section. - -## 1. Publishing a deal - -Charlie heard he could provide storage to people worldwide and earn some tokens, -so he decided to register as a [Storage Provider](../glossary.md). - -```bash -$ storagext-cli --sr25519-key "//Charlie" storage-provider register Charlie -[0xd14d…4b6a] Storage Provider Registered: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, info: Storage Provider Info: { peer_id: 3ZAB4sc5BS, window_post_proof_type: StackedDRGWindow2KiBV1P1, sector_size: _2KiB, window_post_partition_sectors: 2 }, proving_period_start: 21 } -``` - - - -Alice is a [Storage User](../glossary.md#storage-user) and wants to store an image of her lovely Husky (`husky.jpg`) in the Polka Storage [parachain](../glossary.md#parachain). - -Alice knows[^no-cid] that she needs to get a [CID](https://github.com/multiformats/cid) of the image, -so she [uploaded it to the CAR server](../storage-provider-cli/storage.md#upload-a-file) -and received the CID: `bafybeihxgc67fwhdoxo2klvmsetswdmwwz3brpwwl76qizbsl6ypro6vxq`. - -Alice heard somewhere[^no-sp-discovery] in the hallways of her favourite gym that Charlie is a Storage Provider. -She calls him (off-chain), and they negotiate a deal: - -`husky-deal.json` - -```json -[ - { - "piece_cid": "bafybeihxgc67fwhdoxo2klvmsetswdmwwz3brpwwl76qizbsl6ypro6vxq", - "piece_size": 1278, - "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", - "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", - "label": "My lovely Husky (husky.jpg)", - "start_block": 25, - "end_block": 50, - "storage_price_per_block": 1000000000, - "provider_collateral": 12500000000, - "state": "Published" - } -] -``` - -| Name | Value | Description | -| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | -| `piece_cid` | `bafybeihxgc67fwhdoxo2klvmsetswdmwwz3brpwwl76qizbsl6ypro6vxq` | The submitted file's CID | -| `piece_size` | `1278` | The submitted file's size | -| `client` | [`//Alice`](https://docs.substrate.io/reference/glossary/#dev-phrase) or `5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY` | The client's address | -| `provider` | [`//Charlie`](https://docs.substrate.io/reference/glossary/#dev-phrase) or `5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y` | The providers's address | -| `start_block` | `25` | When the deal should start | -| `end_block` | `50` | When the deal should end | -| `storage_price_per_block` | `1_000_000_000` | In [Plancks](../glossary.md#planck), the storage price per block (12 sec) | -| `provider_collateral` | `12_500_000_000` | In [Plancks](../glossary.md#planck), the price paid by the storage provider if they fail to uphold their end of the deal | - -After the negotiation, they need to [add funds to the Pallet](../pallets/market.md#add_balance) and then [publish their intent](../pallets/market.md#publish_storage_deals) so it can be checked by the parachain. -So here they go: - -```bash -# Adding balance to Alice's account -$ storagext-cli --sr25519-key "//Alice" market add-balance 25000000000 -[0xe0bd…06f9] Balance Added: { account: 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY, amount: 25000000000 } -# Adding balance to Charlie's account -$ storagext-cli --sr25519-key "//Charlie" market add-balance 12500000000 -[0x25aa…edd3] Balance Added: { account: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, amount: 12500000000 } -# Publishing their deal -$ storagext-cli --sr25519-key "//Charlie" market publish-storage-deals --client-sr25519-key "//Alice" "@husky-deal.json" -[0xd50b…dee6] Deal Published: { deal_id: 0, provider_account: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, client_account: 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY } -``` - -[^no-cid]: We have not provided a standalone command to generate CID out of the file. The CAR server is a temporary showcase component. -[^no-sp-discovery]: We have not yet implemented Storage Provider Discovery protocol. - -## 2. Committing a deal - -After the deals have been published, the rest is up to Charlie. -If Charlie does not behave appropriately, [pre-commit](../pallets/storage-provider.md#pre_commit_sector) and [prove](../pallets/storage-provider.md#prove_commit_sector) the deal by block 25 (`start_block`), -he will be slashed, and all his funds [(`provider_collateral`)](../glossary.md#collateral) will be gone.[^slash] -So he should do his part! - -`pre-commit-husky.json` - -```json -{ - "sector_number": 1, - "sealed_cid": "bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu", - "deal_ids": [0], - "expiration": 75, - "unsealed_cid": "bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu", - "seal_proof": "StackedDRG2KiBV1P1" -} -``` - -| Name | Value | Description | -| -------------------------------------------------------------------------------------------------------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------- | -| [`sector_number`](../glossary.md#sector) | 1 | The place where `husky.jpg` will be stored. Charlie decided it'll be on his 1st sector. | -| `deal_ids` | `[0]` | A sector can contain multiple deals, but it only contains the first one ever created (id: 0). | -| `expiration` | `75` | The 75th block is 5 minutes after the `end_block`, so the sector expires only after the deal has been terminated. | -| [`sealed_cid`](../glossary.md#commitment-of-replication), [`unsealed_cid`](../glossary.md#commitment-of-data), | multiple | Currently, placeholder values (any CID) since the proof mechanism is a work-in-progress. | -| `seal_proof` | `StackedDRG2KiBV1P1` | Currently, we only accept sector sizes of 2KiB, so this is the only value possible. | - -`prove-commit-husky.json` - -```json -{ - "sector_number": 1, - "proof": "1230deadbeef" -} -``` - -- `proof`: hex string of bytes of the proof, it's WIP, so any non-zero value works. - -```bash -$ storagext-cli --sr25519-key "//Charlie" storage-provider pre-commit "@pre-commit-husky.json" -[0xf2ad…dc3d] Sector Pre-Committed: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, sector_number: Sector Pre-Commit Info: { sector_number: 1, expiration: 75, seal_proof: RegisteredSealProof::StackedDRG2KiBV1P1, unsealed_cid: bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu, sealed_cid: bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu } } -$ storagext-cli --sr25519-key "//Charlie" storage-provider prove-commit "@prove-commit-husky.json" -[0x0743…6fd3] Sector Proven: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, sector_number: 1, partition_number: 0, deadline_idx: 0 } -``` - - - -[^slash]: Wait for 5 minutes and observe a `DealSlashed` [Event](../pallets/market.md#events) being published. - -## 3. Proofs and faults - -### Aside on Deadlines - -There is a little something that Charlie needs to know about: deadlines _(don't we all...)_. - -Each Storage Provider has a Proving Period, a time divided into segments (deadlines). -To simplify, let's say a proving period lasts a day (24 hours), and between the start and end of each hour, there is a segment, just like on a clock. - - - -Now, when Charlie gathers enough data and stores it in a sector, he must keep proving that he has some data. -Charlie can have multiple sectors and he doesn't want to overload the system by proving all of them simultaneously. -So what if each sector got assigned a segment (1 hour) during the day, and Charlie would need to submit proof that he has data roughly on the same hour each day? -That'd work, right? - -So this is what a Proving Period and Deadlines are. -We divide a proving period into deadlines and when we prove commit, we assign a sector to a deadline. -From now on, the sector must be proven periodically and daily during this lifetime. - -`windowed-post.json` - -```json -{ - "deadline": 0, - "partitions": [0], - "proof": { - "post_proof": "2KiB", - "proof_bytes": "1230deadbeef" - } -} -``` - -| Name | Value | Description | -| ------------- | ---------------- | ----------------------------------------------------------------------------- | -| `deadline` | `0` | the deadline index which has been assigned by the Prove Commit | -| `partitions` | `[0]` | the partitions which have been assigned by Prove Commit | -| `post_proof` | "2KiB" | we only support sectors of size 2KiB for now, so it's the only possible value | -| `proof_bytes` | `0x1230deadbeef` | hex string of bytes of the proof, it's WIP, so any non-zero value works | - -Now that the sector has been proven, Charlie must keep confirming that he stores the data. -Charlie's proving period starts at block `21` (as `register-storage-provider` tells us), so the first deadline is between blocks `[21, 31)`, second `[31, 41)`. -That's because there are [2 deadlines](#const-period-deadlines) within a [proving period](#const-proving-period), and a deadline has a [window of 10 blocks](#const-challenge-window). -Charlie got assigned the first deadline, so he waits until block `21` to send the proof that he still stores the data. - -```bash -$ storagext-cli --sr25519-key "//Charlie" storage-provider submit-windowed-post "@windowed-post.json" -[0x3aa1…a12a] Valid PoSt Submitted: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y } -``` - - - -The next proving period starts at block `41`, with the deadline `[41, 51)`, so Charlie must submit his proof on this block. -He knows he won't be able to create a proof, as his hard drives went down, so he reports it. If he didn't report, he'd get slashed. - -`fault-declaration.json` - -```json -[ - { - "deadline": 0, - "partition": 0, - "sectors": [1] - } -] -``` - -```bash -$ storagext-cli --sr25519-key "//Charlie" storage-provider declare-faults "@fault-declaration.json" -[0x49cd…9a88] Faults Declared: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, faults: [Fault Declaration: { deadline: 0, partition: 0, sectors: [1] }] } -``` - - - -Charlie fixed his issues with storage and now wants to declare that he can still provide data in this sector. -If he does this too late (1 minute before the next deadline starts), he won't be able to. - -```bash -$ storagext-cli --sr25519-key "//Charlie" storage-provider declare-faults-recovered "@fault-declaration.json" -Error: Runtime error: Pallet error: StorageProvider::FaultRecoveryTooLate - -Caused by: - Pallet error: StorageProvider::FaultRecoveryTooLate -``` - -If he does it at least a minute before, it succeeds: - -```bash -$ storagext-cli --sr25519-key "//Charlie" storage-provider declare-faults-recovered "@fault-declaration.json" -[0xca6b…875c] Faults Recovered: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, recoveries: [Recovery Declaration: { deadline: 0, partition: 0, sectors: [1] }] } -``` - -And then, at the next deadline, in the next proving period `[41, 51)` he needs to remember to submit windowed PoSt again. - -```bash -$ storagext-cli --sr25519-key "//Charlie" storage-provider submit-windowed-post "@windowed-post.json" -[0x3e46…5636] Valid PoSt Submitted: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y } -``` - -## 4. Reaping the rewards - -After the deal ends (after block `50`), Charlie goes to get his rewards! -First, he settles all of the locked funds, so his balance gets unlocked, and then he withdraws the balance from his Market account to use his DOTs for a new shiny dumbbell. - -```bash -$ storagext-cli --sr25519-key "//Charlie" market settle-deal-payments 0 -[0x1633…d17b] Deals settled: { successful: [Settled Deal { deal_id: 0, provider_account: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, client_account: 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY, amount: 25000000000 }], unsuccessful: [] } -$ storagext-cli --sr25519-key "//Charlie" market withdraw-balance 37500000000 -[0x0197…bdd9] Balance Withdrawn: { account: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, amount: 37500000000 } -``` diff --git a/src/getting-started/index.md b/src/getting-started/index.md index a552a7b..c722024 100644 --- a/src/getting-started/index.md +++ b/src/getting-started/index.md @@ -11,6 +11,13 @@ Before proceeding with the setup, please ensure the host system meets the follow ## Guides -- [*Local Testnet - Polka Storage Parachain*](local-testnet.md) — Covers how to setup a local testnet for the Polka Storage parachain, using Zombienet. -- [*CAR server*](car-server.md) — Covers how to setup and run the CARv2 server, which transforms files into `.car` archives. -- [*Real-world use case demo*](demo.md) — Covers a real-world use case, adding balances, submitting sectors, and more! +- [_Building Polka Storage_](./building/index.md) + - [_Building from source_](./building/source.md) - Covers how to setup the required dependencies on your machine and build components of the project. + - [_Setting up and building with Docker_](./building/docker.md) - Covers how to setup and build the Docker images for the components of the project. +- [_Local Testnet - Polka Storage Parachain_](local-testnet/index.md) — Covers how to setup a local testnet for the Polka Storage parachain, using Zombienet. + - [_Getting funds_](local-testnet/getting-funds.md) - Covers how to get funds into an account that has been generated externally. +- [_Local Testnet - Polka Storage Parachain_](local-testnet/index.md) — Covers how to setup a local testnet for the Polka Storage parachain, using Zombienet. +- [_Launching a Storage Provider_](storage-provider.md) - Covers how to setup a Storage Provider. +- [_Storing a file_](demo-file-store.md) — Covers how to store a file by the Storage Client. + + diff --git a/src/getting-started/local-testnet/getting-funds.md b/src/getting-started/local-testnet/getting-funds.md new file mode 100644 index 0000000..c33a7b5 --- /dev/null +++ b/src/getting-started/local-testnet/getting-funds.md @@ -0,0 +1,60 @@ +# Getting funds + +This document covers getting funds into an account that has been generated externally. + +## Setting up your account + +In this guide we will be covering getting funds into a externally generated account. The recommended way to generate an account is by using the [polkadot.js wallet extension](https://github.com/polkadot-js/extension). + +Please make sure to follow the instructions on how to generate a new account if you have not done so already. +You can read more about creating a Polkadot account using the extension in the following [link](https://support.polkadot.network/support/solutions/articles/65000098878-how-to-create-a-dot-account#How-to-create-an-account-with-the-Polkadot-extension) + +Or you can watch the following video: + + +## Transferring funds + +Make sure to run the local testnet, you can find how to do so in the [local testnet guide](index.md). Once the local testnet is up and running navigate to the polkadot-js web app interface by going to the [default polkadot.js web interface URL](https://polkadot.js.org/apps/?rpc=ws://127.0.0.1:42069). + +> If you have changed the `ws_port` value in the zombienet configuration — `local-testnet.toml`, +> this URL is different and you should change the port accordingly. + +Under the developer tab, navigate to *Sudo*. + +![sudo selection](../../images/transfer-funds/developer-sudo.png) + +Once you are in *Sudo* you should select `balances` from the submit dropdown. + +![balance selection](../../images/transfer-funds/select-balance.png) + +Then, on the box to the right, select `forceSetBalance` from the dropdown. + +![force selection](../../images/transfer-funds/select-force.png) + +Setting the `Id` field to your generated address and the `newFree` field to the amount in [plancks](../../glossary.md#planck), +as shown in the image below. +If your polkadot.js extension is injected into the polkadot.js web interface it will recognize the injection and you can select the desired account. + +
+ +Note that the `forceSetBalance` extrinsic does **NOT** add tokens to an account but rather **sets the balance** to the given amount. + +
+ +![balance forceSetBalance](../../images/transfer-funds/force-set-balance.png) + +Sign and submit your transaction, the caller will automatically be set to Alice, a dev account. + +![sign and submit](../../images/transfer-funds/sign-and-submit.png) + +After the block has been finalized, the balance will show up in the generated account under the accounts tab and you are ready to start using the polka-storage chain with your own account. + +![account balance](../../images/transfer-funds/account-balance.png) diff --git a/src/getting-started/local-testnet.md b/src/getting-started/local-testnet/index.md similarity index 67% rename from src/getting-started/local-testnet.md rename to src/getting-started/local-testnet/index.md index 42de7a8..d05dadb 100644 --- a/src/getting-started/local-testnet.md +++ b/src/getting-started/local-testnet/index.md @@ -8,13 +8,15 @@ Charlie will be our contact point to the parachain network. ## Native Binaries -Our latest releases's binaries are available to download and can be run without additional dependencies. +The binaries for the latest releases are available to download and can be run without any additional dependencies. We support `Linux x86_64` and `MacOS ARM x64`. The commands below will download: - [Relay Chain](https://github.com/paritytech/polkadot-sdk/releases) binaries (`polkadot`, `polkadot-prepare-worker`, `polkadot-execute-worker`), - Polka Storage Parachain binary (`polka-storage-node`), -- [Polka Storage Provider](../storage-provider-cli/index.md) internal node (`polka-storage-provider`), -- [CLI for interacting with the parachain](../storagext-cli/) (`storagext-cli`), +- [Polka Storage Provider](../../storage-provider-cli/server.md) internal node (`polka-storage-provider-server`), +- [Polka Storage Provider Client](../../storage-provider-cli/client/index.md) internal node's RPC client and proving tools (`polka-storage-provider-client`), +- [Storagext CLI](../../storagext-cli/index.md) to interact with a chain by sending extrinsics (`storagext-cli`), +- [Mater CLI](../../mater-cli/index.md) for CARv2 file archive operations (`mater-cli`), - [zombienet](https://paritytech.github.io/zombienet/install.html) to spawn local testnets and orchestrate them (`zombienet`), - Polka Storage Parachain out-of-the-box zombienet's configuration (`polka-storage-testnet.toml`). @@ -23,19 +25,21 @@ We support `Linux x86_64` and `MacOS ARM x64`. The commands below will download: 1. Download the binaries: ```bash -wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-v1.13.0/polkadot -wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-v1.13.0/polkadot-prepare-worker -wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-v1.13.0/polkadot-execute-worker -wget https://s3.eu-central-1.amazonaws.com/polka-storage/linux_x86-64/polka-storage-node -wget https://s3.eu-central-1.amazonaws.com/polka-storage/linux_x86-64/polka-storage-provider -wget https://s3.eu-central-1.amazonaws.com/polka-storage/linux_x86-64/storagext-cli +wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2407-1/polkadot +wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2407-1/polkadot-prepare-worker +wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2407-1/polkadot-execute-worker +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-node-v0.0.0/polka-storage-node-linux-x86 -O polka-storage-node +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-provider-client-v0.1.0/polka-storage-provider-client-linux-x86 -O polka-storage-provider-client +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-provider-server-v0.1.0/polka-storage-provider-server-linux-x86 -O polka-storage-provider-server +wget https://github.com/eigerco/polka-storage/releases/download/storagext-cli-v0.1.0/storagext-cli-linux-x86 -O storagext-cli +wget https://github.com/eigerco/polka-storage/releases/download/mater-cli-v0.1.0/mater-cli-linux-x86 -O mater-cli wget https://github.com/paritytech/zombienet/releases/download/v1.3.106/zombienet-linux-x64 -O zombienet ``` 2. Setup permissions: ```bash -chmod +x zombienet polka-storage-node polka-storage-provider storagext-cli polkadot polkadot-prepare-worker polkadot-execute-worker +chmod +x zombienet polka-storage-node polka-storage-provider-client polka-storage-provider-server storagext-cli mater-cli polkadot polkadot-prepare-worker polkadot-execute-worker ``` 3. Run `zombienet`: @@ -52,20 +56,22 @@ zombienet -p native spawn polka-storage-testnet.toml 1. Download the binaries: ```bash -wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/polkadot -wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/polkadot-prepare-worker -wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/polkadot-execute-worker -wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/polka-storage-node -wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/polka-storage-provider -wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/storagext-cli +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-node-v0.0.0/polkadot-2407-1-macos-arm64 -O polkadot +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-node-v0.0.0/polkadot-2407-1-prepare-worker-macos-arm64 -O polkadot-prepare-worker +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-node-v0.0.0/polkadot-2407-1-execute-worker-macos-arm64 -O polkadot-execute-worker +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-node-v0.0.0/polka-storage-node-macos-arm64 -O polka-storage-node +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-provider-server-v0.1.0/polka-storage-provider-server-macos-arm64 -O polka-storage-provider-server +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-provider-client-v0.1.0/polka-storage-provider-client-macos-arm64 -O polka-storage-provider-client +wget https://github.com/eigerco/polka-storage/releases/download/storagext-cli-v0.1.0/storagext-cli-macos-arm64 -O storagext-cli +wget https://github.com/eigerco/polka-storage/releases/download/mater-cli-v0.1.0/mater-cli-macos-arm64 -O mater-cli wget https://github.com/paritytech/zombienet/releases/download/v1.3.106/zombienet-macos-arm64 -O zombienet ``` 2. Setup permissions & de-quarantine: ```bash -chmod +x zombienet polka-storage-node polka-storage-provider storagext-cli polkadot polkadot-prepare-worker polkadot-execute-worker -xattr -d com.apple.quarantine zombienet polka-storage-node polka-storage-provider storagext-cli polkadot polkadot-prepare-worker polkadot-execute-worker +chmod +x zombienet polka-storage-node polka-storage-provider-server polka-storage-provider-client storagext-cli mater-cli polkadot polkadot-prepare-worker polkadot-execute-worker +xattr -d com.apple.quarantine zombienet polka-storage-node polka-storage-provider-server polka-storage-provider-client storagext-cli mater-cli polkadot polkadot-prepare-worker polkadot-execute-worker ```
@@ -90,7 +96,7 @@ The parachain is also accessible using the Polkadot.js Apps interface by clickin

-Or interact with the chain via [`storagext-cli`](../storagext-cli/index.md), for example: +Or interact with the chain via [`storagext-cli`](../../storagext-cli/index.md), for example: ```bash storagext-cli --sr25519-key "//Alice" storage-provider register Alice @@ -131,7 +137,7 @@ image_pull_policy = "IfNotPresent" [relaychain] chain = "rococo-local" default_args = ["--detailed-log-output", "-lparachain=debug,xcm=trace,runtime=trace"] -default_image = "docker.io/parity/polkadot:v1.13.0" +default_image = "docker.io/parity/polkadot:stable2407-1" [[relaychain.nodes]] name = "alice" @@ -153,12 +159,14 @@ id = 1000 [[parachains.collators]] args = ["--detailed-log-output", "-lparachain=debug,xcm=trace,runtime=trace"] command = "polka-storage-node" -image = "polkadotstorage.azurecr.io/parachain-node:0.1.0" +image = "ghcr.io/eigerco/polka-storage-node:0.0.0" name = "charlie" rpc_port = 42069 validator = true ``` +> For details on this configuration, refer to the [Zombienet Configuration](../../zombienet-config.md) chapter. + 2. Run the Parachain, and spawn the zombienet testnet in the Kubernetes cluster: ``` @@ -222,7 +230,9 @@ zombienet -p kubernetes spawn local-kube-testnet.toml Check if all zombienet pods were started successfully: -`kubectl get pods --all-namespaces` +```bash +kubectl get pods --all-namespaces +```
Click here to show the example output. @@ -252,57 +262,66 @@ The parachain is available through the Polkadot.js Apps interface by clicking on This link will automatically connect to Charlie's node running on a local machine at port `42069`. The port is configured in `local-kube-testnet.toml` under `rpc_port` for Charlie's node. -## Zombienet Configuration Breakdown - -During the setup either download a file in the third step of [Linux](#linux-x86_64)/[MacOS](#macos-arm) — `polka-storage-testnet.toml` — -or copy it from the first step of [Running the parachain](#running-the-parachain). - -### Similarities - -The two files share most of the contents, so we'll start by covering their similarities. -For more details refer to the [`zombienet` documentation](https://paritytech.github.io/zombienet/network-definition-spec.html): - -#### `relaychain` +## Checking the logs -| Name | Description | -| ----------------- | --------------------------------------------- | -| `chain` | The relaychain name | -| `default_args` | The default arguments passed to the `command` | -| `default_command` | The default command to run the relaychain | -| `nodes` | List of tables defining the nodes to run | - -##### `nodes` - -| Name | Description | -| ----------- | -------------------------------------- | -| `name` | The node name | -| `validator` | Whether the node is a validator or not | - -#### `parachains` +At the end of the `zombienet` output you should see a table like so: +``` +┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ +│ Node Information │ +├──────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ Name │ charlie │ +├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ Direct Link │ https://polkadot.js.org/apps/?rpc=ws://127.0.0.1:42069#/explorer │ +├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ Prometheus Link │ http://127.0.0.1:44955/metrics │ +├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ Log Cmd │ tail -f │ +│ │ /tmp/nix-shell.gQQj4Y/zombie-bcb786e1748ff0a6becd28289e1f70b9_-677866-G8ea9Qqs65DB/charlie.log │ +├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ Parachain ID │ 1000 │ +├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ ChainSpec Path │ /tmp/nix-shell.gQQj4Y/zombie-bcb786e1748ff0a6becd28289e1f70b9_-677866-G8ea9Qqs65DB/1000-rococo-lo… │ +└──────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘ +``` -A list of tables defining multiple parachains, in our case, we only care for our own parachain. +We strongly recommend you check the logs for the collator (in this case Charlie), using a text editor of your choice. -| Name | Description | -| --------------- | ------------------------------------------------------------- | -| `cumulus_based` | Whether to use `cumulus` based generation | -| `id` | The parachain ID, we're using `1000` as a placeholder for now | -| `collators` | List of tables defining the collators | +> If the **Log Cmd** is shortened with `...`, try to search for the folder using the zombienet namespace, available at the top of the table. +> For example: +> ```bash +> $ ls /tmp | grep zombie-bcb786e1748ff0a6becd28289e1f70b9 +> ``` -##### `collators` +After finding the `charlie.log` file, you should `grep` for errors relating to the extrinsic you just ran, +for example, if you ran a `market` extrinsic, like `add-balance`, you would run: -| Name | Description | -| ----------- | ---------------------------------------- | -| `args` | The arguments passed to the `command` | -| `command` | The command to run the collator | -| `name` | The collator name | -| `validator` | Whether the collator is also a validator | +```bash +$ grep "ERROR.*runtime::market" charlie.log +``` -### Differences +Or, more generally: -The difference between them lies in the usage of container configurations: +```bash +$ grep ".*runtime::" charlie.log +``` -| Name | Description | -| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `image_pull_policy` | Defines when `zombienet` should pull an image; read more about it in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy) | -| `image` | Defines which image to pull | -| `ws_port`/`rpc_port` | Depending on the type of configuration (Native or Kubernetes), this variable sets the port for the collator RPC service | +Where `LOG_LEVEL` is one of: +* `DEBUG` +* `INFO` +* `WARN` +* `ERROR` + +And the extrinsic pallet, is one of: +* `storage_provider` +* `market` +* `proofs` + +> **Tip:** if you are running into the `AllProposalsInvalid` error, +> try searching for `insane deal` in the logs, you should find the cause faster! +> +> For example: +> ```bash +> $ grep "insane deal" -B 1 charlie.log +> 2024-11-14 13:24:24.019 ERROR tokio-runtime-worker runtime::market: [Parachain] deal duration too short: 100 < 288000 +> 2024-11-14 13:24:24.019 ERROR tokio-runtime-worker runtime::market: [Parachain] insane deal: idx 0, error: ProposalError::DealDurationOutOfBounds +> ``` diff --git a/src/getting-started/storage-provider.md b/src/getting-started/storage-provider.md new file mode 100644 index 0000000..b6f3d8a --- /dev/null +++ b/src/getting-started/storage-provider.md @@ -0,0 +1,107 @@ +# Launching the Storage Provider + +> This guide assumes you have read the [Building](./building/index.md) +> and the [Local Testnet - Polka Storage](./local-testnet/index.md) guides +> and have a running testnet to connect to. + +Setting up the Storage Provider doesn't have a lot of science, but isn't automatic either! +In this guide, we'll cover how to get up and running with the Storage Provider. + +## Generating the PoRep Parameters + +First and foremost, to allow the Storage Provider to generate [PoRep](https://docs.filecoin.io/basics/the-blockchain/proofs#proof-of-replication-porep) proofs, +we need to first generate their parameters, we do that with the following command: + +```bash +$ polka-storage-provider-client proofs porep-params +Generating params for 2KiB sectors... It can take a couple of minutes ⌛ +Generated parameters: +/home/user/polka-storage/2KiB.porep.params +/home/user/polka-storage/2KiB.porep.vk +/home/user/polka-storage/2KiB.porep.vk.scale +``` + +As advertised, the command has generated the following files: + +* `2KiB.porep.params` — The PoRep parameters +* `2KiB.porep.vk` — The verifying key +* `2KiB.porep.vk.scale` — The verifying key, encoded in SCALE format + +## Registering the Storage Provider + +> If you encounter errors while running extrinsics, check the parachain logs. +> Refer to the [Checking the logs](./local-testnet/index.md#checking-the-logs) section under the +> [Local Testnet - Polka Storage Provider](./local-testnet/index.md) chapter. + +Logically, if you want to participate in the network, you need to register. +To do so, you need to run one of the following commands: + +```bash +storagext-cli --sr25519-key storage-provider register "" +storagext-cli --ed25519-key storage-provider register "" +storagext-cli --ecdsa-key storage-provider register "" +``` + + +Where `` has been replaced accordingly to its key type. +`` can be anything as it is currently used as a placeholder. *For example:* +``` +storagext-cli --sr25519-key "//Charlie" storage-provider register "placeholder" +``` + +After registering, there is one more thing to be done, to be able to verify proofs in local testnet. +We need to set the global verifying key in the network, so it's compatible with the proving parameters: + +```bash +storagext-cli --sr25519-key "//Charlie" proofs set-porep-verifying-key @2KiB.porep.vk.scale +``` + +> Additionally, you will need to add some balance to your Polka Storage escrow account, like so: +> ``` +> $ storagext-cli --sr25519-key "//Charlie" market add-balance 12500000000 +> [0x809d…8f10] Balance Added: { account: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, amount: 12500000000 } +> ``` +> You can use other balance values! There's a minimum though — `1_000_000_000` (without the `_`). + +And you're ready! + +## Launching the server 🚀 + +Similarly to the previous steps, here too you'll need to run a command. +The following is the *minimal* command: + +```bash +polka-storage-provider-server \ + --seal-proof 2KiB \ + --post-proof 2KiB \ + --porep-parameters \ + --X-key +``` + +Where `--X-key ` matches the key type you used to register yourself with the network, in the previous step. For example: +```bash +polka-storage-provider-server \ + --seal-proof 2KiB \ + --post-proof 2KiB \ + --porep-parameters "2KiB.porep.params" \ + --sr25519-key "//Charlie" +``` + +Note that currently, `--seal-proof` and `--post-proof` only support `2KiB`. + +`` is the resulting `*.porep.params` file from the [first steps](#generating-the-porep-parameters), +in this case, `2KiB.porep.params`. + +When ran like this, the server will assume a random directory for the database and the storage, however, +you can change that through the `--database-directory` and `--storage-directory`, respectively, +if the directory does not exist, it will be created. + +You can also change the parachain node address it connects to, +by default, the server will try to connect to `ws://127.0.0.1:42069`, +but you can change this using the `--node-url` flag. + +Finally, you can change the listening addresses for the RPC and HTTP services, +they default to `127.0.0.1:8000` and `127.0.0.1:8001` respectively, +and can be changed using the flags `--rpc-listen-address` and `--upload-listen-address`. + +For more information on the available flags, refer to the [`server` chapter](../storage-provider-cli/server.md). diff --git a/src/images/architecture/collator_overview.svg b/src/images/architecture/collator_overview.svg new file mode 100644 index 0000000..c1644f4 --- /dev/null +++ b/src/images/architecture/collator_overview.svg @@ -0,0 +1,172 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/images/architecture/pallets_overview.png b/src/images/architecture/pallets_overview.png new file mode 100644 index 0000000..b12f400 Binary files /dev/null and b/src/images/architecture/pallets_overview.png differ diff --git a/src/images/architecture/storage-provider-pipeline.png b/src/images/architecture/storage-provider-pipeline.png new file mode 100644 index 0000000..3244560 Binary files /dev/null and b/src/images/architecture/storage-provider-pipeline.png differ diff --git a/src/images/architecture/storage-provider-server.png b/src/images/architecture/storage-provider-server.png new file mode 100644 index 0000000..b3773d8 Binary files /dev/null and b/src/images/architecture/storage-provider-server.png differ diff --git a/src/images/architecture/storage-provider-server.svg b/src/images/architecture/storage-provider-server.svg new file mode 100644 index 0000000..70316de --- /dev/null +++ b/src/images/architecture/storage-provider-server.svg @@ -0,0 +1,109 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/images/architecture/storage_provider_overview.svg b/src/images/architecture/storage_provider_overview.svg new file mode 100644 index 0000000..1fc14b1 --- /dev/null +++ b/src/images/architecture/storage_provider_overview.svg @@ -0,0 +1,122 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/images/architecture/system_overview.png b/src/images/architecture/system_overview.png new file mode 100644 index 0000000..9e68843 Binary files /dev/null and b/src/images/architecture/system_overview.png differ diff --git a/src/images/architecture/system_overview.svg b/src/images/architecture/system_overview.svg new file mode 100644 index 0000000..a19f5c4 --- /dev/null +++ b/src/images/architecture/system_overview.svg @@ -0,0 +1,116 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/images/background/challenge_protocol.png b/src/images/background/challenge_protocol.png new file mode 100644 index 0000000..86d4384 Binary files /dev/null and b/src/images/background/challenge_protocol.png differ diff --git a/src/images/background/merkle_tree_comparison.png b/src/images/background/merkle_tree_comparison.png new file mode 100644 index 0000000..2bba1e6 Binary files /dev/null and b/src/images/background/merkle_tree_comparison.png differ diff --git a/src/images/background/merkle_tree_storage_provider.png b/src/images/background/merkle_tree_storage_provider.png new file mode 100644 index 0000000..89c63cc Binary files /dev/null and b/src/images/background/merkle_tree_storage_provider.png differ diff --git a/src/images/background/merkle_tree_verifier.png b/src/images/background/merkle_tree_verifier.png new file mode 100644 index 0000000..cac72e9 Binary files /dev/null and b/src/images/background/merkle_tree_verifier.png differ diff --git a/src/images/polkadot.svg b/src/images/polkadot.svg new file mode 100644 index 0000000..9d5bad6 --- /dev/null +++ b/src/images/polkadot.svg @@ -0,0 +1,14 @@ + + + diff --git a/src/images/showcase/readme.md b/src/images/showcase/README.md similarity index 59% rename from src/images/showcase/readme.md rename to src/images/showcase/README.md index 48e3ac2..0198916 100644 --- a/src/images/showcase/readme.md +++ b/src/images/showcase/README.md @@ -3,12 +3,12 @@ The tools used for the showcase are [asciinema](https://asciinema.org/), [asciin Steps to record a new asciinema session: 1. Install asciinema or use the docker image -2. Start the Local Testnet - Polka Storage Parachain -3. Update the `cli_basic.sh` with the commands you want to showcase +2. Check the README.md of the specific showcase for any additional steps +3. Update the `showcase.sh` with the commands you want to showcase 4. Start the asciinema recording - `asciinema-automation --timeout 300 ./cli_basic.sh prod.cast` + `asciinema-automation --timeout 300 ./showcase.sh showcase.cast` 5. Convert the `.cast` to `.gif` - `agg prod.cast prod.gif` + `agg showcase.cast showcase.gif` diff --git a/src/images/showcase/cli_basic.gif b/src/images/showcase/cli_basic.gif deleted file mode 100644 index 09295f8..0000000 Binary files a/src/images/showcase/cli_basic.gif and /dev/null differ diff --git a/src/images/showcase/cli_basic.sh b/src/images/showcase/cli_basic.sh deleted file mode 100644 index a804a86..0000000 --- a/src/images/showcase/cli_basic.sh +++ /dev/null @@ -1,60 +0,0 @@ -# This script demonstrates the basic workflow of a storage provider. It is used -# by the asciinema_automation to automate the video session. It includes -# registering a provider, adding balances, publishing storage deals, -# pre-committing sectors, and proving commitments. It uses the storagext-cli -# tool to interact with the network. - -# The script expecs the storagext-cli tool to be in the current directory. - -# mean of gaussian delay between key strokes, default to 50ms -#$ delay 20 - -./storagext-cli --sr25519-key "//Alice" \ - market add-balance 1000000000000 -#$ expect Balance Added - -./storagext-cli --sr25519-key "//Charlie" \ - market add-balance 1000000000000 -#$ expect Balance Added - -./storagext-cli --sr25519-key "//Alice" \ - storage-provider register alice -#$ expect Storage Provider Registered - -./storagext-cli --sr25519-key "//Alice" \ - market publish-storage-deals --client-sr25519-key "//Charlie" ' -[ - { - "piece_cid": "bafk2bzacecg3xxc4f2ql2hreiuy767u6r72ekdz54k7luieknboaakhft5rgk", - "piece_size": 1337, - "client": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", - "provider": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", - "label": "Awesome piece", - "start_block": 100, - "end_block": 200, - "storage_price_per_block": 15, - "provider_collateral": 2000, - "state": "Published" - } -]' -#$ expect Deal Published - -./storagext-cli --sr25519-key "//Alice" \ - storage-provider pre-commit ' -{ - "sector_number": 0, - "sealed_cid": "bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu", - "deal_ids": [0], - "expiration": 200, - "unsealed_cid": "bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu", - "seal_proof": "StackedDRG2KiBV1P1" -}' -#$ expect Sector Pre-Committed - -./storagext-cli --sr25519-key "//Alice" \ - storage-provider prove-commit ' -{ - "sector_number": 0, - "proof": "64756D6D792070726F6F66" -}' -#$ expect Sector Proven diff --git a/src/images/showcase/client_upload/README.md b/src/images/showcase/client_upload/README.md new file mode 100644 index 0000000..c5ebf38 --- /dev/null +++ b/src/images/showcase/client_upload/README.md @@ -0,0 +1,12 @@ +### Prerequisites + +Script expects `mater-cli`, `polka-storage-provider-client`, `polka-storage-provider-server`, `storagext-cli` in the current directory. + +1. Start zombienet and call required extrinsics + - Setup Alices balances `./storagext-cli --sr25519-key //Alice market add-balance 250000000000` + - Setup Charlies balances `./storagext-cli --sr25519-key //Charlie market add-balance 250000000000` + - Register Charlie as a storage provider `./storagext-cli --sr25519-key "//Charlie" storage-provider register "peer_id"` +2. Generate proving params + - `./polka-storage-provider-client proofs porep-params` +3. Start storage provider server + - `./polka-storage-provider-server --sr25519-key //Charlie --seal-proof "2KiB" --post-proof "2KiB" --porep-parameters 2KiB.porep.params` diff --git a/src/images/showcase/client_upload/data.txt b/src/images/showcase/client_upload/data.txt new file mode 100644 index 0000000..5855f49 --- /dev/null +++ b/src/images/showcase/client_upload/data.txt @@ -0,0 +1,158 @@ +1234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +1234567891 +1234567891 +1234567891 +1234567891 +1234567891 +1234567891 +123 \ No newline at end of file diff --git a/src/images/showcase/client_upload/deal.json b/src/images/showcase/client_upload/deal.json new file mode 100644 index 0000000..5fb8cc6 --- /dev/null +++ b/src/images/showcase/client_upload/deal.json @@ -0,0 +1,12 @@ +{ + "piece_cid": "baga6ea4seaqbfhdvmk5qygevit25ztjwl7voyikb5k2fqcl2lsuefhaqtukuiii", + "piece_size": 2048, + "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", + "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", + "label": "custom label", + "start_block": 200, + "end_block": 250, + "storage_price_per_block": 500, + "provider_collateral": 1250, + "state": "Published" +} \ No newline at end of file diff --git a/src/images/showcase/client_upload/showcase.gif b/src/images/showcase/client_upload/showcase.gif new file mode 100644 index 0000000..909b100 Binary files /dev/null and b/src/images/showcase/client_upload/showcase.gif differ diff --git a/src/images/showcase/client_upload/showcase.sh b/src/images/showcase/client_upload/showcase.sh new file mode 100644 index 0000000..ce14faf --- /dev/null +++ b/src/images/showcase/client_upload/showcase.sh @@ -0,0 +1,45 @@ +# This script demonstrates the process of uploading data to Polka Storage, +# creating and signing a storage deal, and publishing it on the network. It is +# used by the asciinema_automation to automate the video session. +# +# The script performs the following steps: +# 1. Converts input data to CAR format +# 2. Generates CommP (Piece CID) for the data +# 3. Creates a storage deal JSON +# 4. Signs the storage deal +# 5. Proposes the deal and obtains a Deal CID +# 6. Uploads the data to the storage provider +# 7. Publishes the signed deal on the network + +# mean of gaussian delay between key strokes, default to 50ms +#$ delay 20 + +./mater-cli convert -q data.txt data.car +#$ expect + +./polka-storage-provider-client proofs commp data.car +#$ expect cid + +DEAL_JSON="$(cat ./deal.json)" +echo "$DEAL_JSON" + +SIGNED_DEAL_JSON="$(./polka-storage-provider-client sign-deal --sr25519-key //Alice "$DEAL_JSON")" +#$ expect + +echo "$SIGNED_DEAL_JSON" +#$ expect client_signature + +DEAL_CID="$(./polka-storage-provider-client propose-deal "$DEAL_JSON")" +#$ expect + +echo "$DEAL_CID" +#$ expect bagaaieravcqzkt2ilbdghlw3metiwuuqklfq2udxiubsghyj47ha3wuogppq + +curl -X PUT -F "upload=@data.txt" "http://localhost:8001/upload/$DEAL_CID" +#$ expect baga6ea4seaqbfhdvmk5qygevit25ztjwl7voyikb5k2fqcl2lsuefhaqtukuiii + +# wait instructions change the time between instructions, default to 80ms +#$ wait 1000 + +./polka-storage-provider-client publish-deal "$SIGNED_DEAL_JSON" +#$ expect 0 diff --git a/src/images/storage-provider/sector_activation.mermaid b/src/images/storage-provider/sector_activation.mermaid new file mode 100644 index 0000000..2b267a4 --- /dev/null +++ b/src/images/storage-provider/sector_activation.mermaid @@ -0,0 +1,8 @@ +sequenceDiagram + Storage Provider->>+Randomness Pallet: get_randomness(seal_block) + Storage Provider->>+Storage Provider Pallet: pre_commit + Storage Provider Pallet->>+Market Pallet: verify_deals + Storage Provider->>+Randomness Pallet: get_porep_randomness(precommit_block) + Storage Provider->>+Storage Provider Pallet: prove_commit_sectors + Storage Provider Pallet->>+Proofs Pallet: verify_porep + Storage Provider Pallet->>+Market Pallet: activate_deals \ No newline at end of file diff --git a/src/images/storage-provider/sector_activation.svg b/src/images/storage-provider/sector_activation.svg index 4434efb..a51f543 100644 --- a/src/images/storage-provider/sector_activation.svg +++ b/src/images/storage-provider/sector_activation.svg @@ -1,3 +1,3 @@ -Market PalletStorage Provider PalletStorage Providerpre_commit_sectorverify_dealsprove_commit_sectoractivate_deals +Proofs PalletMarket PalletStorage Provider PalletRandomness PalletStorage ProviderProofs PalletMarket PalletStorage Provider PalletRandomness PalletStorage Providerget_randomness(seal_block)pre_commitverify_dealsget_porep_randomness(precommit_block)prove_commit_sectorsverify_porepactivate_deals \ No newline at end of file diff --git a/src/images/transfer-funds/account-balance.png b/src/images/transfer-funds/account-balance.png new file mode 100644 index 0000000..d6f9e52 Binary files /dev/null and b/src/images/transfer-funds/account-balance.png differ diff --git a/src/images/transfer-funds/developer-sudo.png b/src/images/transfer-funds/developer-sudo.png new file mode 100644 index 0000000..9be3519 Binary files /dev/null and b/src/images/transfer-funds/developer-sudo.png differ diff --git a/src/images/transfer-funds/force-set-balance.png b/src/images/transfer-funds/force-set-balance.png new file mode 100644 index 0000000..ec8f397 Binary files /dev/null and b/src/images/transfer-funds/force-set-balance.png differ diff --git a/src/images/transfer-funds/select-balance.png b/src/images/transfer-funds/select-balance.png new file mode 100644 index 0000000..b403f6a Binary files /dev/null and b/src/images/transfer-funds/select-balance.png differ diff --git a/src/images/transfer-funds/select-force.png b/src/images/transfer-funds/select-force.png new file mode 100644 index 0000000..fb16f66 Binary files /dev/null and b/src/images/transfer-funds/select-force.png differ diff --git a/src/images/transfer-funds/sign-and-submit.png b/src/images/transfer-funds/sign-and-submit.png new file mode 100644 index 0000000..0c371ba Binary files /dev/null and b/src/images/transfer-funds/sign-and-submit.png differ diff --git a/src/introduction.md b/src/introduction.md index 7020a97..b44c0b0 100644 --- a/src/introduction.md +++ b/src/introduction.md @@ -4,38 +4,70 @@ Welcome to the Polka Storage project! This project aims to deliver a Polkadot-native system parachain for data storage. -Since the Referendum approval, we've been busy developing the parachain, -and this is our deliverable for Phase 1, composed of: - -- Filecoin actor ports: - - [Storage Provider](./pallets/storage-provider.md) — excluding proving mechanisms - - [Market](./pallets/market.md) -- [CAR file conversion server](./storage-provider-cli/storage.md) -- Dedicated CLIs - - [`storage-provider-cli`](./storage-provider-cli/storage.md) to generate keys and test our CARv2 Rust implementation! - - [`storagext-cli`](./storagext-cli/index.md) (shown below) to take the parachain for a spin! +Since the Referendum approval, we've been busy developing the deliverables for Phase 2. + +**For [**Phase 2**](https://polkadot.polkassembly.io/referenda/1150), we have implemented:** + +- Storage Provider Pallet + - [`terminate_sectors`](./architecture/pallets/storage-provider.md#terminate_sectors) + - `process_early_terminations` (internal) + - Batch support + - [Pre-commit](./architecture/pallets/storage-provider.md#pre_commit_sectors) + - [Prove-commit](./architecture/pallets/storage-provider.md#prove_commit_sectors) +- [Storage Provider Server](./architecture/polka-storage-provider-server.md) + - [File upload](./getting-started/demo-file-store.md) + - [Proving Pipeline](./architecture/polka-storage-provider-server.md#sealing-pipeline) + - Pre-commit + - PoRep +- On-chain proof validation + - Proof of Replication + - Proof of Spacetime + + + +We also present a complete [real-world scenario](./getting-started/demo-file-store.md) in which a [Storage Provider](./glossary.md#storage-provider) and a [Storage User](./glossary.md#storage-user) negotiate a deal, perform all the steps necessary to start the storage and then receive rewards (or punishments) for making it happen. + +**The Polka Storage project currently provides:** + +Dedicated CLIs + +- [Polka Storage Provider CLI](./storage-provider-cli/index.md) + - [`polka-storage-provider-server`](./storage-provider-cli/server.md) to launch the Storage Provider server. + - [`polka-storage-provider-client`](./storage-provider-cli/client/index.md) to manage the wallet, propose & publish deals and do proof demos. +- [`mater-cli`](./mater-cli/index.md) to convert or extract CARv2 files. +- [`storagext-cli`](./storagext-cli/index.md) to interact **directly** with the parachain — watch out, this is a low-level tool! + +Filecoin actor ports: + +- [Storage Provider](./architecture/pallets/storage-provider.md) +- [Market](./architecture/pallets/market.md) +- [Proofs](./architecture/pallets/proofs.md) +- [Randomness](./architecture/pallets/randomness.md) +

Polka Storage CLI tooling showcase + src="images/showcase/client_upload/showcase.gif" + alt="Polka Storage Client Upload">

-The following on-chain logic has been implemented: +**As a refresher, for [Phase 1](https://polkadot.polkassembly.io/referenda/494), we implemented the following:** - Keeping track of [Storage Providers](./glossary.md#storage-provider), -- [Publishing](./pallets/market.md#publish_storage_deals) Market Deals on-chain, -- [Investing](./pallets/market.md#add_balance) tokens into the Storage Market, -- [Receiving](./pallets/market.md#settle_deal_payments) funds after completing a deal, -- [Commiting](./pallets/storage-provider.md#pre_commit_sector) to the Storage and [Proving](./pallets/storage-provider.md#prove_commit_sector) the storage, -- [Declaring](./pallets/storage-provider.md#prove_commit_sector) failures to deliver committed storage and [Recovering](./pallets/storage-provider.md#declaring-storage-faults-recovered) from it, -- [Continuously proving](./pallets/storage-provider.md#submit_windowed_post) that the promise of storage has been kept up [PoSt proof](./glossary.md#proofs), -- [Punishing](./pallets/storage-provider.md#events) for failing to provide storage. +- [Publishing](./architecture/pallets/market.md#publish_storage_deals) Market Deals on-chain, +- [Investing](./architecture/pallets/market.md#add_balance) tokens into the Storage Market, +- [Receiving](./architecture/pallets/market.md#settle_deal_payments) funds after completing a deal, +- [Committing](./architecture/pallets/storage-provider.md#pre_commit_sectors) to the Storage and [Proving](./architecture/pallets/storage-provider.md#prove_commit_sectors) the storage, +- [Declaring](./architecture/pallets/storage-provider.md#prove_commit_sectors) failures to deliver committed storage and [Recovering](./architecture/pallets/storage-provider.md#declaring-storage-faults-recovered) from it, +- [Continuously proving](./architecture/pallets/storage-provider.md#submit_windowed_post) that the promise of storage has been kept up [PoSt proof](./glossary.md#proofs), +- [Terminating](./architecture/pallets/storage-provider.md#terminate_sectors) sectors by the storage provider, +- [Punishing](./architecture/pallets/storage-provider.md#events) for failing to provide storage. -We present a complete [real-world scenario](./getting-started/demo.md) in which a [Storage Provider](./glossary.md#storage-provider) and a [Storage User](./glossary.md#storage-user) negotiate a deal, perform all the steps necessary to start the storage and then receive rewards (or punishments) for making it happen. +We present a demo on how to [store a file](./getting-started/demo-file-store.md), where a [Storage Provider](./glossary.md#storage-provider) and a [Storage User](./glossary.md#storage-user) negotiate a deal and perform all the steps necessary to start the file storage. We cover the details behind proving a file in a [separate demo](./getting-started/demo-file-prove.md). -More information available about the project's genesis in: +**More information available about the project's genesis in:** -- OpenGov Referendum — +- OpenGov Referendum - Part 1 — +- OpenGov Referendum - Part 2 — - Research Report — - Polkadot Forum News Post — diff --git a/src/mater-cli/index.md b/src/mater-cli/index.md new file mode 100644 index 0000000..0f5fef1 --- /dev/null +++ b/src/mater-cli/index.md @@ -0,0 +1,84 @@ +# Mater CLI + + + +The Mater CLI is used by storage clients to convert files to the CARv2 format and extract CARv2 content. + +> Currently, the `mater-cli` only supports the CARv2 format. +> However, _the `mater` library has full support for CARv1_. + +To learn more about the CAR format, please refer to the official specifications: + +- CARv1 — +- CARv2 — + +## `convert` + +The convert command converts a file to CARv2 format. + +`mater-cli convert [OUTPUT_PATH]` + +| Argument | Description | +| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| `` | Path to input file | +| `[OUTPUT_PATH]` | Optional path to output CARv2 file. If no output path is given it will store the `.car` file in the same location as the input file. | +| `-q`/`--quiet` | If enabled, only the resulting CID will be printed. | +| `--overwrite` | If enabled, the output will overwrite any existing files. | + +### Example + +```bash +$ mater-cli convert random1024.piece +Converted examples/random1024.piece and saved the CARv2 file at examples/random1024.car with a CID of bafkreidvyofebclo4kny43vpoe5kejg3mqtpq2eemaojzyvlwikwdvusxy +``` + +You can verify the output file using [`go-car`](https://github.com/ipld/go-car): + +```bash +$ car inspect examples/random1024.car +Version: 2 +Characteristics: 00000000000000000000000000000000 +Data offset: 51 +Data (payload) length: 1121 +Index offset: 1172 +Index type: car-multihash-index-sorted +Roots: bafkreidvyofebclo4kny43vpoe5kejg3mqtpq2eemaojzyvlwikwdvusxy +Root blocks present in data: Yes +Block count: 1 +Min / average / max block length (bytes): 1024 / 1024 / 1024 +Min / average / max CID length (bytes): 36 / 36 / 36 +Block count per codec: + raw: 1 +CID count per multihash: + sha2-256: 1 +``` + +## `extract` + +Convert a CARv2 file to its original format. + +`mater-cli extract [OUTPUT_PATH]` + +| Argument | Description | +| --------------- | ------------------------------------------------------------------------------------------------------------------------------ | +| `` | Path to CARv2 file | +| `[OUTPUT_PATH]` | Optional path to output file. If no output path is given it will remove the extension and store the file in the same location. | + +### Example + +```bash +$ mater-cli extract examples/random1024.car +Successfully converted CARv2 file examples/random1024.car and saved it to to examples/random1024 +``` + +Conversly, you can also extract files generated using `car`: + +```bash +# --no-wrap is necessary since mater does not perform wrapping +$ car create --no-wrap -f examples/random1024.go.car examples/random1024.piece +``` + +```bash +$ cargo run -r --bin mater-cli extract examples/random1024.go.car +Successfully converted CARv2 file examples/random1024.go.car and saved it to to examples/random1024.go +``` diff --git a/src/pallets/index.md b/src/pallets/index.md deleted file mode 100644 index e4e0742..0000000 --- a/src/pallets/index.md +++ /dev/null @@ -1,52 +0,0 @@ -# Polka Storage pallets - -- [`storage-provider`](storage-provider.md) - A pallet that manages storage providers and their associated data. -- [`market`](market.md) - A pallet that handles the storage market operations. - -## Overview - -The Polka Storage parachain is all about making storage deals. Let us go over how a deal is done! - -Before anything else, [Storage Providers](../glossary.md#storage-provider) need to register themselves with the [Storage Provider Pallet](./storage-provider.md) — they can do so using the [`register_storage_provider`](./storage-provider.md#register_storage_provider) extrinsic. - -Storage Provider registration - -Now that storage providers can be registered in the storage provider pallet, we need to add some balance to both the -[Storage User's](../glossary.md#storage-user) and the Provider's accounts, -which is done using the Market's [`add_balance`](./market.md#add_balance) extrinsic. - -Adding balance to Market accounts - -Afterwards, storage users and providers negotiate data storage deals off-chain. -Once a deal between the two parties is reached, the client can sign the deal and send it to the storage provider for publishing -— the storage provider will then publish the signed deal using the [`publish_storage_deals`](market.md#publish_storage_deals) extrinsic. - -After publishing, the funds allocated for the deal will be moved from `free` to `locked`, and they can no longer be withdrawn until the deal has ended. - -Publishing storage deals - -At this point, the remaining responsibility is shifted to the storage provider, which needs to activate the deal; -to do so, the storage provider first needs to [pre-commit](./storage-provider.md#pre_commit_sector) the deal's sectors, -and then [prove](./storage-provider.md#prove_commit_sector) they stored the sectors -— these two steps are done using the [`pre_commit_sector`](./storage-provider.md#pre_commit_sector) and [`prove_commit_sector`](./storage-provider.md#prove_commit_sector) extrinsics. - -Verification is done and reported to the Market pallet to terminate the deal and apply penalties to the storage provider -(remove and burn its collateral — i.e. `locked` funds) if they fail to activate the deal on time and return the funds to the client. - -Deal activation - -Suppose the deal has been completed successfully or is **Active**. -In that case, the storage provider is now required to periodically submit proofs that they're still storing the user's data -— the storage provider does this by calculating a proof and submitting it using [`submit_windowed_post`](./storage-provider.md#submit_windowed_post). - -Proving the data is still stored - -Finally, storage providers can then settle deal payments to receive their fair share for keeping the user's data safe — using the [`settle_deal_payments`](./market.md#settle_deal_payments) extrinsic. - -Settling deal payments - -Putting it all together, we get the following: - -The described flow - - diff --git a/src/storage-provider-cli/client/index.md b/src/storage-provider-cli/client/index.md new file mode 100644 index 0000000..518b990 --- /dev/null +++ b/src/storage-provider-cli/client/index.md @@ -0,0 +1,116 @@ +# client + +We cover the commands provided by the `polka-storage-provider-client` CLI tool. + +## `wallet` + +The `wallet` command is a thin wrapper over the [`subkey`](https://docs.substrate.io/reference/command-line-tools/subkey/) utility provided by Polkadot. + +More information available on the [`wallet`](./wallet.md) page. + +## `info` + +The `info` command retrieves information about the storage provider it connects to. + +```bash +$ polka-storage-provider-client info --rpc-server-url "http://127.0.0.1:8000" +{ + "start_time": "2024-11-06T11:29:06.058967136Z", + "address": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", + "seal_proof": "StackedDRG2KiBV1P1", + "post_proof": "StackedDRGWindow2KiBV1P1" +} +``` + +## `propose-deal` + +The `propose-deal` command sends an *unsigned* deal to the storage provider, +if the storage provider accepts the deal, a CID will be returned, +that CID can then be used to upload a file to the storage provider — +for details on this process, refer to the [File Upload chapter](../../getting-started/demo-file-store.md). + +> For the current MVP, the storage provider accepts all valid deals! + +```bash +$ DEAL_TO_PROPOSE='{ + "piece_cid": "baga6ea4seaqj527iqfb2kqhy3tmpydzroiigyaie6g3txai2kc3ooyl7kgpeipi", + "piece_size": 2048, + "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", + "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", + "label": "", + "start_block": 200, + "end_block": 250, + "storage_price_per_block": 500, + "provider_collateral": 1250, + "state": "Published" +}' +# when we omit the `--rpc-server-address` it defaults to "http://127.0.0.1:8000" +$ polka-storage-provider-client propose-deal "$DEAL_TO_PROPOSE" +bagaaieradsfmawozrmgjwxosarexpg7w7ytoe7xw2c63hv6svdc5hpucqo3a +``` + +## `sign-deal` + +The `sign-deal` commands takes a deal like the one passed to [`propose-deal`](#propose-deal) and signs it using the passed key, +the returned deal can then be used with [`publish-deal`](#publish-deal) to send a deal for publishing. +*This command does not call out to the network.* + +```bash +$ DEAL_TO_SIGN='{ + "piece_cid": "baga6ea4seaqj527iqfb2kqhy3tmpydzroiigyaie6g3txai2kc3ooyl7kgpeipi", + "piece_size": 2048, + "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", + "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", + "label": "", + "start_block": 200, + "end_block": 250, + "storage_price_per_block": 500, + "provider_collateral": 1250, + "state": "Published" +}' +$ polka-storage-provider-client sign-deal --sr25519-key "//Charlie" "$DEAL_TO_SIGN" +{ + "deal_proposal": { + "piece_cid": "baga6ea4seaqj527iqfb2kqhy3tmpydzroiigyaie6g3txai2kc3ooyl7kgpeipi", + "piece_size": 2048, + "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", + "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", + "label": "", + "start_block": 200, + "end_block": 250, + "storage_price_per_block": 500, + "provider_collateral": 1250, + "state": "Published" + }, + "client_signature": { + "Sr25519": "32809cd5b53fa3c2e977f77c4e2189dee230b8773946cf94a704f8af19c578289c11ad256b56146195cfc5d7bb8f670003e4575e133f799f19696495046ed58f" + } +} +``` + +## `publish-deal` + +The `publish-deal` command effectively publishes the deal, its input is a deal signed using [`sign-deal`](#sign-deal), +and the output is the on-chain deal ID. + +```bash +$ SIGNED_DEAL='{ + "deal_proposal": { + "piece_cid": "baga6ea4seaqj527iqfb2kqhy3tmpydzroiigyaie6g3txai2kc3ooyl7kgpeipi", + "piece_size": 2048, + "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", + "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", + "label": "", + "start_block": 200, + "end_block": 250, + "storage_price_per_block": 500, + "provider_collateral": 1250, + "state": "Published" + }, + "client_signature": { + "Sr25519": "32809cd5b53fa3c2e977f77c4e2189dee230b8773946cf94a704f8af19c578289c11ad256b56146195cfc5d7bb8f670003e4575e133f799f19696495046ed58f" + } +}' +$ polka-storage-provider-client publish-deal "$SIGNED_DEAL" +0 +``` diff --git a/src/storage-provider-cli/client/proofs.md b/src/storage-provider-cli/client/proofs.md new file mode 100644 index 0000000..4edb913 --- /dev/null +++ b/src/storage-provider-cli/client/proofs.md @@ -0,0 +1,15 @@ +# `proofs` + +The following subcommands are contained under `proofs`. + +> These are advanced commands and only useful for demo purposes. +> This functionality is covered in the server by the [pipeline](../../architecture/polka-storage-provider-server.md#sealing-pipeline). + +| Name | Description | +| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | +| `calculate-piece-commitment` | Calculate a piece commitment for the provided data stored at the a given path | +| `porep-params` | Generates PoRep verifying key and proving parameters for zk-SNARK workflows (prove commit) | +| `post-params` | Generates PoSt verifying key and proving parameters for zk-SNARK workflows (submit windowed PoSt) | +| `porep` | Generates PoRep for a piece file. Takes a piece file (in a CARv2 archive, unpadded), puts it into a sector (temp file), seals and proves it | +| `post` | Creates a PoSt for a single sector | + diff --git a/src/storage-provider-cli/wallet.md b/src/storage-provider-cli/client/wallet.md similarity index 94% rename from src/storage-provider-cli/wallet.md rename to src/storage-provider-cli/client/wallet.md index a255b58..08a355c 100644 --- a/src/storage-provider-cli/wallet.md +++ b/src/storage-provider-cli/client/wallet.md @@ -30,7 +30,7 @@ Keys shown on this page are, by default, not secure! Do not use them in producti Generate a new random key to interact with the Polka Storage parachain: ```bash -> polka-storage-provider wallet generate +> polka-storage-provider-client wallet generate Secret phrase: offer payment boost boy manage car asset lock cousin mountain vehicle setup Network ID: substrate Secret seed: 0xfe36ee692552b0ce54de06ce4f5cc152fe2fa808cb40f58c81168bc1237208bb @@ -43,7 +43,7 @@ Secret phrase: offer payment boost boy manage car asset lock cousin mounta The password may be added interactively, using `--password-interactive` flag: ```bash -> polka-storage-provider wallet generate --password-interactive +> polka-storage-provider-client wallet generate --password-interactive Key password: