diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 92c0a26..abd4d22 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -1,18 +1,43 @@ # Summary -- [Introduction](./introduction.md) +[Introduction](./introduction.md) +[Background](./background.md) + +# Guides + +- [Architecture](./architecture/index.md) + - [Polka Storage Provider Server](./architecture/polka-storage-provider-server.md) + - [Pallets](./architecture/pallets/index.md) + - [Market](./architecture/pallets/market.md) + - [Storage Provider](./architecture/pallets/storage-provider.md) + - [Proofs](./architecture/pallets/proofs.md) + - [Randomness](./architecture/pallets/randomness.md) - [Getting Started](./getting-started/index.md) - - [Local Testnet - Polka Storage Parachain](./getting-started/local-testnet.md) - - [CAR server](./getting-started/car-server.md) - - [Real-world use case demo](./getting-started/demo.md) + - [Building](./getting-started/building/index.md) + - [From Source](./getting-started/building/source.md) + - [Using Docker](./getting-started/building/docker.md) + - [Local Testnet - Polka Storage Parachain](./getting-started/local-testnet/index.md) + - [Getting funds](./getting-started/local-testnet/getting-funds.md) + - [Launching the Storage Provider](./getting-started/storage-provider.md) + - [Proving a file](./getting-started/demo-file-prove.md) + - [Storing a file](./getting-started/demo-file-store.md) + +# Tools + - [Polka Storage Provider CLI](./storage-provider-cli/index.md) - - [`storage`](./storage-provider-cli/storage.md) - - [`wallet`](./storage-provider-cli/wallet.md) + - [`server`](./storage-provider-cli/server.md) + - [`client`](./storage-provider-cli/client/index.md) + - [`wallet`](./storage-provider-cli/client/wallet.md) + - [`proofs`](./storage-provider-cli/client/proofs.md) - [Storagext CLI](./storagext-cli/index.md) - [`market`](./storagext-cli/market.md) - [`storage-provider`](./storagext-cli/storage-provider.md) + - [`proofs`](./storagext-cli/proofs.md) + - [`randomness`](./storagext-cli/randomness.md) - [`system`](./storagext-cli/system.md) -- [Pallets](./pallets/index.md) - - [Market](./pallets/market.md) - - [Storage Provider](./pallets/storage-provider.md) -- [Glossary](./glossary.md) +- [Mater CLI](./mater-cli/index.md) +- [Zombienet Configuration](./zombienet-config.md) + +--- + +[Glossary](./glossary.md) diff --git a/src/architecture/index.md b/src/architecture/index.md new file mode 100644 index 0000000..8f5e446 --- /dev/null +++ b/src/architecture/index.md @@ -0,0 +1,45 @@ +# Architecture Overview + +The Polka Storage parachain is, just like other parachains, composed of collators that receive extrinsics calls, +and through them perform state transitions. + +## System Overview + +![](../images/architecture/system_overview.png) + +From left to right, we have validators (represented by a single node as only one validates blocks at a time), +collators and the storage providers. + +The validators are handled by Polkadot itself — i.e. who gets to check the validity proofs is randomly selected by the network. + +The collators execute our parachain runtime and process extrinsic calls from the storage providers — such as proof of storage submissions. + +The storage providers are independent of the collators and are controlled by people like you, who provide storage to the system. +Storage management is left to the storage providers, being responsible to keep their physical system in good shape to serve clients. +We do provide an implementation of the storage provider, you can read more about it in the [Polka Storage Provider Server chapter](./polka-storage-provider-server.md). + +## Pallets Overview + +![](../images/architecture/pallets_overview.png) + +We've been focusing on implementing the core functionality by developing the market, storage provider, proofs and randomness pallets. + +The market pallet handles all things related to deal payments and slashing, +being informed by the storage provider when deals haven't been proven and applying slashing in those cases. +The storage provider handles the registering of storage providers and the proof submission, +the latter is checked inside the collator's WASM runtime, using the proofs pallet. +Finally, the proofs pallet makes use of randomness for challenges, ensuring the storage providers can't cheat the system. + +For a deeper dive on the pallets, you can read the [Pallets chapter](./pallets/index.md). + +## Resources on Parachains + +Reading: +* [Parachains' Protocol Overview](https://wiki.polkadot.network/docs/learn-parachains-protocol) +* [The Path of a Parachain Block](https://polkadot.com/blog/the-path-of-a-parachain-block) + +Videos: +* [Introduction to Polkadot, Parachains, and Substrate](https://www.youtube.com/live/gT-9r1bcVHY?si=dmCJyWB5w2NY1bnu&t=1670) +* [The Path of a Parachain Block - Joe Petrowski](https://www.youtube.com/watch?v=vRsBlVELQEo) +* [The Path of a Parachain Block on Polkadot and Kusama Network](https://www.youtube.com/watch?v=m0vxqWwFfDs) + diff --git a/src/architecture/pallets/index.md b/src/architecture/pallets/index.md new file mode 100644 index 0000000..30ff44e --- /dev/null +++ b/src/architecture/pallets/index.md @@ -0,0 +1,55 @@ +# Polka Storage pallets + +- [`storage-provider`](storage-provider.md) - A pallet that manages storage providers and their associated data. +- [`market`](market.md) - A pallet that handles the storage market operations. +- [`proofs`](proofs.md) - A pallet responsible for verifying [PoRep](../../glossary.md#porep) and [PoSt](../../glossary.md#post). +- [`randomness`](randomness.md) - A pallet providing randomness source for blocks, mainly used by Proofs. + +## Overview + +The Polka Storage parachain is all about making storage deals. Let us go over how a deal is done! + +Before anything else, [Storage Providers](../../glossary.md#storage-provider) need to register themselves with the [Storage Provider Pallet](./storage-provider.md) — they can do so using the [`register_storage_provider`](./storage-provider.md#register_storage_provider) extrinsic. + +Storage Provider registration + +Now that storage providers can be registered in the storage provider pallet, we need to add some balance to both the +[Storage User's](../../glossary.md#storage-user) and the Provider's accounts, +which is done using the Market's [`add_balance`](./market.md#add_balance) extrinsic. + +Adding balance to Market accounts + +Afterwards, storage users and providers negotiate data storage deals off-chain. +Once a deal between the two parties is reached, the client can sign the deal and send it to the storage provider for publishing +— the storage provider will then publish the signed deal using the [`publish_storage_deals`](market.md#publish_storage_deals) extrinsic. + +After publishing, the funds allocated for the deal will be moved from `free` to `locked`, and they can no longer be withdrawn until the deal has ended. + +Publishing storage deals + +At this point, the remaining responsibility is shifted to the storage provider, which needs to activate the deal. +First, the storage provider needs to call `get_randomness` from the [Randomness Pallet](./randomness.md) in order to create a replica and +[pre-commit](./storage-provider.md#pre_commit_sectors) the deal's sectors. +The sealing and pre-committing takes some time, after that the storage provider needs to fetch yet another randomness seed to create a proof. +Subsequently, they [prove](./storage-provider.md#prove_commit_sectors) they stored the sectors by calling [`prove_commit_sectors`](./storage-provider.md#prove_commit_sectors) extrinsics. + +Verification is done via the [Proofs Pallet](./proofs.md) and reported to the Market pallet to terminate the deal and apply penalties to the storage provider +(remove and burn its collateral — i.e. `locked` funds) if they fail to activate the deal on time and return the funds to the client. + +Deal activation + +Suppose the deal has been completed successfully or is **Active**. +In that case, the storage provider is now required to periodically submit proofs that they're still storing the user's data +— the storage provider does this by calculating a proof and submitting it using [`submit_windowed_post`](./storage-provider.md#submit_windowed_post). + +Proving the data is still stored + +Finally, storage providers can then settle deal payments to receive their fair share for keeping the user's data safe — using the [`settle_deal_payments`](./market.md#settle_deal_payments) extrinsic. + +Settling deal payments + +Putting it all together, we get the following: + +The described flow + + diff --git a/src/pallets/market.md b/src/architecture/pallets/market.md similarity index 86% rename from src/pallets/market.md rename to src/architecture/pallets/market.md index 1439c57..504db7a 100644 --- a/src/pallets/market.md +++ b/src/architecture/pallets/market.md @@ -18,7 +18,7 @@ The purpose of the pallet is to manage storage deals between storage market part Market Pallet is tightly coupled with [Storage Provider Pallet](./storage-provider.md) because it's a source of truth for deals. Storage Provider Pallet cannot exist without deal information from Market Pallet. -## Extrinsics\* +## Extrinsics\* ### `add_balance` @@ -32,7 +32,7 @@ The reserved amount will be considered `free` until it is used in a deal when #### Example -Using the `storagext-cli` to add 1000000000[^existential_deposit] [Plancks](../glossary.md#planck) to Alice's account with the following command[^add_balance]: +Using the `storagext-cli` to add 1_000_000_000[^existential_deposit] [Plancks](../../glossary.md#planck) to Alice's account with the following command[^add_balance]: ```bash storagext-cli --sr25519-key "//Alice" market add-balance 1000000000 @@ -42,14 +42,14 @@ storagext-cli --sr25519-key "//Alice" market add-balance 1000000000 This value is the minimum amount due to Polkadot's existential deposit. More information available in: . -[^add_balance]: Read more about the `add-balance` command in [_Storagext CLI/Subcommand `market`/`add-balance`_](../storagext-cli/market.md#add-balance) +[^add_balance]: Read more about the `add-balance` command in [_Storagext CLI/Subcommand `market`/`add-balance`_](../../storagext-cli/market.md#add-balance). ### `withdraw_balance` Withdraws funds from the Storage Market. The funds will be withdrawn from the `free` balance, meaning that the `amount` must be -lesser than or equal to `free` and greater than 0 (\\({free} \ge {amount} \gt 0\\)). +less than or equal to `free` and greater than 0 (\\({free} \ge {amount} \gt 0\\)). | Name | Description | Type | | -------- | -------------------------- | ---------------- | @@ -57,13 +57,13 @@ lesser than or equal to `free` and greater than 0 (\\({free} \ge {amount} \gt 0\ #### Example -Using the `storagext-cli` to withdraw 10000 [Plancks](../glossary.md#planck) from Alice's `free` balance using the following command[^withdraw_balance]: +Using the `storagext-cli` to withdraw 10000 [Plancks](../../glossary.md#planck) from Alice's `free` balance using the following command[^withdraw_balance]: ```bash storagext-cli --sr25519-key "//Alice" market withdraw-balance 10000 ``` -[^withdraw_balance]: Read more about the `withdraw-balance` command in [_Storagext CLI/Subcommand `market`/`withdraw-balance`_](../storagext-cli/market.md#withdraw-balance) +[^withdraw_balance]: Read more about the `withdraw-balance` command in [_Storagext CLI/Subcommand `market`/`withdraw-balance`_](../../storagext-cli/market.md#withdraw-balance). ### `publish_storage_deals` @@ -73,11 +73,11 @@ This extrinsic _must_ be called by a storage provider. | Name | Description | Type | | ------------------ | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `proposal` | Specific deal proposal, a JSON object | JSON object, specified in the [next section](#deal-proposal-components) | +| `proposal` | Specific deal proposal, a JSON object | JSON object, specified in the [deal proposal components section](#deal-proposal-components) | | `client_signature` | Client signature of this specific deal proposal | [`MultiSignature`](https://paritytech.github.io/polkadot-sdk/master/sp_runtime/enum.MultiSignature.html), meaning a 64-byte array for Sr25519 and Ed25519 signatures and 65-byte array for ECDSA signatures | The `client_signature`, as the name indicates, is generated by the client by signing the deal proposal with their private key — -the [`storagext-cli`](../storagext-cli/index.md) does this for the user automatically[^warn]. +the [`storagext-cli`](../../storagext-cli/index.md) does this for the user automatically[^warn]. This signature ensures that the storage provider cannot forge a deal with an arbitrary client. The type of signature is dependent on the key the signer has, currently supported key types are Sr25519, ECDSA and Ed25519. @@ -94,8 +94,8 @@ _This step corresponds to the "sign & send proposal" step in the [deal overview] | `label` | Arbitrary client chosen label | String, with a maximum length of 128 characters | | `start_block` | Block number on which the deal should start | Positive integer | | `end_block` | Block number on which the deal should end | Positive integer, `end_block > start_block` | -| `storage_price_per_block` | Price for the storage specified per block[^storage_price_per_block] | Positive integer, in [Plancks](../glossary.md#planck) | -| `provider_collateral` | Collateral which is slashed if the deal fails | Positive integer, in [Plancks](../glossary.md#planck) | +| `storage_price_per_block` | Price for the storage specified per block[^storage_price_per_block] | Positive integer, in [Plancks](../../glossary.md#planck) | +| `provider_collateral` | Collateral which is slashed if the deal fails | Positive integer, in [Plancks](../../glossary.md#planck) | | `state` | Deal state. Can only be set to `Published` | String | See the [original Filecoin specification](https://spec.filecoin.io/#section-systems.filecoin_markets.onchain_storage_market.storage_deal_flow) for details. @@ -149,9 +149,9 @@ and is **not final**; it is a testing tool. -[^storage_price_per_block]: The formula to calculate the total price is as follows: \\[total\\\_price = (end\\\_block - start\\\_block) * storage\\_price\\_per\\_block\\] -[^publish_storage_deals]: Read more about the `publish-storage-deals` command in [_Storagext CLI/Subcommand `market`/`publish-storage-deals`_](../storagext-cli/market.md#publish-storage-deals) [^warn]: Take into account that the CLI is currently for demo purposes, the authors are aware that the command isn't safe since it requires the private keys of both parties. +[^storage_price_per_block]: The formula to calculate the total price is as follows: \\\[total\\\_price = (end\\\_block - start\\\_block) \* storage\\\_price\\\_per\\\_block\\]. +[^publish_storage_deals]: Read more about the `publish-storage-deals` command in [_Storagext CLI/Subcommand `market`/`publish-storage-deals`_](../../storagext-cli/market.md#publish-storage-deals). ### `settle_deal_payments` @@ -174,7 +174,7 @@ Using the `storagext-cli` to settle deal payments for IDs 97, 1010, 1337 and 420 storagext-cli --sr25519-key "//Alice" market settle-deal-payments 97 1010 1337 42069 ``` -[^settle_deal_payments]: Read more about the `settle-deal-payments` command in [_Storagext CLI/Subcommand `market`/`settle-deal-payments`_](../storagext-cli/market.md#settle-deal-payments) +[^settle_deal_payments]: Read more about the `settle-deal-payments` command in [_Storagext CLI/Subcommand `market`/`settle-deal-payments`_](../../storagext-cli/market.md#settle-deal-payments) ## Events @@ -197,9 +197,9 @@ The Market Pallet emits the following events: - `DealsSettled` - Published after the `settle_deal_payments` extrinsic is called. Indicates which deals were successfully and unsuccessfully settled. - `successful` - List of deal IDs that were settled - `unsuccessful` - List of deal IDs with the corresponding errors -- `DealSlashed` - Is emitted when some deal expired +- `DealSlashed` - Is emitted when some deal expired. - `deal_id` - Deal ID that was slashed -- `DealTerminated` - Is emitted it indicates that the deal was voluntarily or involuntarily terminated. +- `DealTerminated` - A deal was voluntarily or involuntarily terminated. - `deal_id` - Terminated deal ID - `client` - SS58 address of the storage client - `provider` - SS58 address of the storage provider @@ -210,7 +210,7 @@ The Market Pallet actions can fail with following errors: - `InsufficientFreeFunds` - Market participants do not have enough free funds. - `NoProposalsToBePublished` - `publish_storage_deals` was called with an empty list of `deals`. -- `ProposalsNotPublishedByStorageProvider` - Is returned when calling `publish_storage_deals` and the deals in a list are not published by the same storage provider. +- `ProposalsPublishedByIncorrectStorageProvider` - Is returned when calling `publish_storage_deals` and the deals in a list are not published by the same storage provider. - `AllProposalsInvalid` - `publish_storage_deals` call was supplied with a list of `deals` which are all invalid. - `DuplicateDeal` - There is more than one deal with this ID in the Sector. - `DealNotFound` - Tried to activate a deal that is not in the system. @@ -227,9 +227,9 @@ The Market Pallet actions can fail with following errors: ## Constants -| Name | Description | Value | -| ------------------ | ----------------------------------------------------------------------------- | ----------- | -| `MaxDeals` | How many deals can be published in a single batch of `publish_storage_deals`. | 128 | -| `MaxDealsPerBlock` | Maximum deals that can be scheduled to start at the same block. | 128 | -| `MinDealDuration` | Minimum time an activated deal should last. | 5 Minutes | -| `MaxDealDuration` | Maximum time an activated deal should last. | 180 Minutes | +| Name | Description | Value | +| ------------------ | ----------------------------------------------------------------------------- | ------------------------- | +| `MaxDeals` | How many deals can be published in a single batch of `publish_storage_deals`. | 128 | +| `MaxDealsPerBlock` | Maximum deals that can be scheduled to start at the same block. | 128 | +| `MinDealDuration` | Minimum time an activated deal should last. | 5 Minutes (50 Blocks) | +| `MaxDealDuration` | Maximum time an activated deal should last. | 180 Minutes (1800 Blocks) | diff --git a/src/architecture/pallets/proofs.md b/src/architecture/pallets/proofs.md new file mode 100644 index 0000000..cc3ed2c --- /dev/null +++ b/src/architecture/pallets/proofs.md @@ -0,0 +1,93 @@ +# Proofs Pallet + +## Table of Contents + +- [Overview](#overview) +- [Usage](#usage) +- [Extrinsics](#extrinsics) + - [`set_porep_verifying_key`](#set_porep_verifying_key) + - [`set_post_verifying_key`](#set_post_verifying_key) +- [Events](#events) +- [Errors](#errors) + +## Overview + +The `Proofs Pallet` handles all the logic related to verifying [PoRep](../../glossary.md#porep) and [PoSt](../../glossary.md#post) proofs on-chain. +It's called by [`Storage Provider Pallet`](./storage-provider.md) when verifying proofs during the extrinsics [`prove_commit_sectors`](./storage-provider.md#prove_commit_sectors) +and [`submit_windowed_post`](./storage-provider.md#submit_windowed_post). The Pallet **DOES NOT** expose any extrinsic for proofs verification, it only implements a trait that can be [coupled to other pallets](https://education.web3.foundation/docs/Substrate/section8/pallet-coupling). + +To verify the proofs properly it needs to have the verifying keys parameters set for the sector size via [`set_porep_verifying_key`](#set_porep_verifying_key) and [`set_post_verifying_key`](#set_post_verifying_key). + +## Usage + +This pallet can only be directly used via the trait `primitives_proofs::ProofVerification`. However, for the trait to work and not fail with `Error::MisingPoRepVerifyingKey`/`Error::MissingPoStVerifingKey`, the verifying keys need to be set via extrinsics `set_porep_verifying_key`/`set_post_verifying_key`. + +Ideally, users shouldn't worry about it, as it will be set by the governance during a trusted setup procedure and then Storage Providers will download the proof generation parameters. However, in the MVP phase, those keys need to be set with the extrinsics after starting a testnet. + +Verifying Keys are set for a Sector Size once and then shared across all proof verifications. +Currently, the network only supports 2KiB sector sizes, so parameters need to be generated and set for it. + +## Extrinsics + +### `set_porep_verifying_key` + +Verifying Key is a set of shared parameters used for zk-SNARK proof verification. It can be generated via [`polka-storage-provider-client proofs porep-params`](../../storage-provider-cli/client/proofs.md#porep-params) command. The verifying key used in the verification must match the proving parameters used in the proof generation. + +The extrinsic sets the verifying key received in the SCALE-encoded format and then uses it for all the subsequent verification. +Verifying Key is used to verify every PoRep proof across the network. + +| Name | Description | Type | +| --------------- | ------------------------------------------------------------- | -------------------------------------- | +| `verifying_key` | shared set of parameters used for zk-SNARK proof verification | SCALE encoded bytes of a Verifying Key | + +#### Example + +Setting a verifying key from the [^account] `//Alice` account where proof is stored in the `./2KiB.porep.vk.scale` file. + +```bash +storagext-cli --sr25519-key "//Alice" proofs set-porep-verifying-key 2KiB.vk.scale +``` + +[^account]: Note that in the MVP every account can set a Verifying Key. It's a risky operation that can halt the entire network, because if verifying key changes, Storage Providers needs to update their generating parameters as well. + +### `set_post_verifying_key` + +Verifying Key is a set of shared parameters used for zk-SNARK proof verification. It can be generated via [`polka-storage-provider-client proofs post-params`](../../storage-provider-cli/client/proofs.md#post-params) command. The verifying key used in the verification must match proving parameters used in the proof generation. + +The extrinsic sets the verifying key received in the SCALE-encoded format and then uses it for all the subsequent verification. +Verifying Key is used to verify every PoSt proof across the network. + +| Name | Description | Type | +| --------------- | ------------------------------------------------------------- | -------------------------------------- | +| `verifying_key` | shared set of parameters used for zk-SNARK proof verification | SCALE encoded bytes of a Verifying Key | + +#### Example + +Setting a verifying key from the [^account] `//Alice` account where proof is stored in the `./2KiB.post.vk.scale` file. + +```bash +storagext-cli --sr25519-key "//Alice" proofs set-post-verifying-key 2KiB.vk.scale +``` + +[^account]: Note that in the MVP every account can set a Verifying Key. It's a risky operation that can halt the entire network, because if verifying key changes, Storage Providers needs to update their generating parameters as well. + +## Events + +The Proofs Pallet emits the following events: + +- `PoRepVerifyingKeyChanged` - PoRep verifying key has been changed. + - `who` - SS58 address of the caller. +- `PoStVerifyingKeyChanged` - PoSt verifying key has been changed. + - `who` - SS58 address of the caller. + +## Errors + +The Proofs Pallet actions can fail with the following errors: + +- `InvalidVerifyingKey` - supplied Verifying Key was not in the valid format and could not be deserialized. +- `InvalidPoRepProof` - PoRep proof could not be verified, it was not created for the given replica window. +- `InvalidPoStProof` - PoSt proof could not be verified, it was not created for the given sector. +- `MissingPoRepVerifyingKey` - tried to verify PoRep proof, but the PoRep verifying key was not set previously with the [`set_porep_verifying_key`](#set_post_verifying_key) extrinsic. +- `MissingPoStVerifyingKey` - tried to verify PoSt proof, but the PoSt verifying key was not set previously with the [`set_post_verifying_key`](#set_post_verifying_key) extrinsic. +- `Conversion` - PoRep/PoSt Proof/VerifyingKey are in an invalid format and cannot be deserialized. + diff --git a/src/architecture/pallets/randomness.md b/src/architecture/pallets/randomness.md new file mode 100644 index 0000000..9dd5d28 --- /dev/null +++ b/src/architecture/pallets/randomness.md @@ -0,0 +1,48 @@ +# Randomness Pallet + +## Table of Contents + +- [Randomness Pallet](#randomness-pallet) + - [Table of Contents](#table-of-contents) + - [Overview](#overview) + - [Usage](#usage) + - [Extrinsics](#extrinsics) + - [Events](#events) + - [Errors](#errors) + - [Constants](#constants) + +## Overview + +The randomness pallet saves a random seed for each block when it's finalized and allows retrieval of this randomness at a later time. +There is a limitation - the randomness is available only after the 81st block of the chain, due to randomness predictability earlier. +Currently, the seeds are used for the sealing pipeline's pre-commit and prove commit, in other words generating a replica and proving a sector. + +## Usage + +This pallet exposes the interface to get randomness on-chain for a certain block via the trait `primitives_proofs::Randomness` +or chain state query `pallet_randomness:SeedsMap`. +Note that, you can only get a randomness for the `current_block - 1` and depending on the configuration, the old randomness seed will be removed after the associated block has passed. + +## Extrinsics + +The pallet does not expose any extrinsics. + +## Events + +The pallet does not emit any events. + +## Errors + +The Randomness Pallet actions can fail with the following errors: + +- `SeedNotAvailable` - the seed for the given block number is not available, which means the randomness pallet has not gathered randomness for this block yet. + +## Constants + +The Storage Provider Pallet has the following constants: + +| Name | Description | Value | +| ----------------- | ----------------------------------------------------------------- | ------- | +| `CleanupInterval` | Clean-up interval specified in number of blocks between cleanups. | 1 Day | +| `SeedAgeLimit` | The number of blocks after which the seed is cleaned up. | 30 Days | + diff --git a/src/pallets/storage-provider-extra.md b/src/architecture/pallets/storage-provider-extra.md similarity index 100% rename from src/pallets/storage-provider-extra.md rename to src/architecture/pallets/storage-provider-extra.md diff --git a/src/pallets/storage-provider.md b/src/architecture/pallets/storage-provider.md similarity index 61% rename from src/pallets/storage-provider.md rename to src/architecture/pallets/storage-provider.md index 7950e6d..0ee5971 100644 --- a/src/pallets/storage-provider.md +++ b/src/architecture/pallets/storage-provider.md @@ -7,11 +7,12 @@ - [Declaring storage faults and recoveries](#declaring-storage-faults-and-recoveries) - [Extrinsics](#extrinsics) - [`register_storage_provider`](#register_storage_provider) - - [`pre_commit_sector`](#pre_commit_sector) - - [`prove_commit_sector`](#prove_commit_sector) + - [`pre_commit_sectors`](#pre_commit_sectors) + - [`prove_commit_sectors`](#prove_commit_sectors) - [`submit_windowed_post`](#submit_windowed_post) - [`declare_faults`](#declare_faults) - [`declare_faults_recovered`](#declare_faults_recovered) + - [`terminate_sectors`](#terminate_sectors) - [Events](#events) - [Errors](#errors) - [Pallet constants](#pallet-constants) @@ -19,26 +20,24 @@ ## Overview The `Storage Provider Pallet` handles the creation of storage providers and facilitates storage providers and clients in creating storage deals. -Storage providers must provide the [Proof of Space-time (PoSt)](../glossary.md#post) and the [Proof of Replication (PoRep)](../glossary.md#porep) -to the `Storage Provider Pallet` to prevent the pallet from imposing penalties on storage providers through [slashing](#storage-fault-slashing). +Storage providers must provide the [Proof of Space-time (PoSt)](../../glossary.md#post) and the [Proof of Replication (PoRep)](../../glossary.md#porep) +to the `Storage Provider Pallet` to prevent the pallet from imposing penalties on storage providers through [slashing](../../glossary.md#slashing). ## Usage ### Declaring storage faults and recoveries Faulty sectors are subject to penalties. To minimize said penalties, the storage provider should declare any sector for which they cannot -generate a [PoSt](../glossary.md#post) as faulty, this will mask said sectors in future deadlines, minimizing the suffered penalties. +generate a [PoSt](../../glossary.md#post) as faulty, this will mask said sectors in future deadlines, minimizing the suffered penalties. A storage provider must declare the sector as faulty [**before**](#fault-declaration-cutoff) the challenge window. Through the [`declare_faults`](#declare_faults) and [`declare_faults_recovered`](#declare_faults_recovered) extrinsics the storage provider can declare sectors as faulty or recovered[^recovered]. -Declaring faults and recoveries +Declaring faults and recoveries [^recovered]: Recovered sectors still require being proven before they can become fully active again. -Substrate pallet hooks execute actions when certain conditions are met. - Substrate pallet hooks execute some actions when certain conditions are met. We use these hooks — when a block finalizes — to check if storage providers are up to date with their proofs. If a storage provider fails to submit proof on time, the Storage Provider pallet will signal the Market pallet to penalize the storage provider. @@ -65,21 +64,22 @@ Registering a storage provider with keypair `//Alice` and peer ID `alice` with t storagext-cli --sr25519-key "//Alice" storage-provider register alice ``` -[^register_storage_provider]: Read more about the `register` command in [_Storagext CLI/Subcommand `storage-provider`/`register`_](../storagext-cli/storage-provider.md#register) +[^register_storage_provider]: Read more about the `register` command in [_Storagext CLI/Subcommand `storage-provider`/`register`_](../../storagext-cli/storage-provider.md#register) -### `pre_commit_sector` +### `pre_commit_sectors` After publishing a deal, the storage provider needs to pre-commit the sector information to the chain. Sectors are not valid after pre-commit. The sectors need to be proven first. +The pre-commit extrinsic takes in an array of the following values: -| Name | Description | Type | -| --------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------- | -| `seal_proof` | Seal proof type this storage provider is using [^note] | String, currently only `StackedDRGWindow2KiBV1P1` is available | -| `sector_number` | The sector number that is being pre-committed | Positive integer | -| `sealed_cid` | [Commitment of replication](../glossary.md#commitment-of-replication) | Hex string of the sealed CID bytes | -| `deal_ids` | Deal IDs to be pre-committed, from `publish_storage_deals` | Array of integers | -| `expiration` | Expiration block of the pre-committed sector | Positive integer | -| `unsealed_cid` | Commitment of data [sector sealing](../glossary.md#commitment-of-data) | Hex string of the unsealed CID bytes | +| Name | Description | Type | +| --------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------- | +| `seal_proof` | Seal proof type this storage provider is using [^note] | String, currently only `StackedDRGWindow2KiBV1P1` is available | +| `sector_number` | The sector number that is being pre-committed | Positive integer | +| `sealed_cid` | [Commitment of replication](../../glossary.md#commitment-of-replication) | Hex string of the sealed CID bytes | +| `deal_ids` | Deal IDs to be pre-committed, from `publish_storage_deals` | Array of integers | +| `expiration` | Expiration block of the pre-committed sector | Positive integer | +| `unsealed_cid` | Commitment of data [sector sealing](../../glossary.md#commitment-of-data) | Hex string of the unsealed CID bytes |
Sectors are not valid after pre-commit. The sectors need to be proven first. @@ -87,9 +87,9 @@ Sectors are not valid after pre-commit. The sectors need to be proven first. [^note]: Only one seal-proof type supported at the moment, `2KiB`. -#### Example +#### Example -Storage provider `//Alice` pre-committing[^pre_commit_sector] a sector number 1, with a single deal ID 0. +Storage provider `//Alice` pre-committing[^pre-committing] a sector number 1, with a single deal ID 0. ```bash storagext-cli --sr25519-key "//Alice" storage-provider pre-commit @pre-commit-sector.json @@ -108,22 +108,23 @@ Where `pre-commit-sector.json` is a file with contents similar to: } ``` -[^pre_commit_sector]: Read more about the `pre-commit` command in [_Storagext CLI/Subcommand `storage-provider`/`pre-commit`_](../storagext-cli/storage-provider.md#pre-commit) +[^pre-committing]: Read more about the `pre-commit` command in [_Storagext CLI/Subcommand `storage-provider`/`pre-commit`_](../../storagext-cli/storage-provider.md#pre-commit) -### `prove_commit_sector` +### `prove_commit_sectors` -After pre-committing some new sectors the storage provider needs to supply a [Proof-of-Replication](../glossary.md#commitment-of-replication) for these sectors [^note]. +After pre-committing some new sectors the storage provider needs to supply a [Proof-of-Replication](../../glossary.md#commitment-of-replication) for these sectors [^note]. +The prove-commit extrinsic takes in an array of the following values: -| Name | Description | Type | -| --------------- | -------------------------------------------------------------------- | ----------------------------- | -| `sector_number` | The sector number that is being prove-committed | Positive integer | -| `proof` | The [proof of replication](../glossary.md#commitment-of-replication) | Hex string of the proof bytes | +| Name | Description | Type | +| --------------- | ----------------------------------------------------------------------- | ----------------------------- | +| `sector_number` | The sector number that is being prove-committed | Positive integer | +| `proof` | The [proof of replication](../../glossary.md#commitment-of-replication) | Hex string of the proof bytes | [^note]: At the moment, any proof of non-zero length is accepted for PoRep. -#### Example +#### Example -This example follows up on the pre-commit example. Storage provider `//Alice` is proven committing[^prove_commit_sector] sector number 1. +This example follows up on the pre-commit example. Storage provider `//Alice` is proven committing[^prove_commit_sectors] sector number 1. ```bash storagext-cli --sr25519-key "//Alice" storage-provider prove-commit @prove-commit-sector.json @@ -138,11 +139,11 @@ Where `prove-commit-sector.json` is a file with contents similar to: } ``` -[^prove_commit_sector]: Read more about the `prove-commit` command in [_Storagext CLI/Subcommand `storage-provider`/`prove-commit`_](../storagext-cli/storage-provider.md#prove-commit) +[^prove_commit_sectors]: Read more about the `prove-commit` command in [_Storagext CLI/Subcommand `storage-provider`/`prove-commit`_](../../storagext-cli/storage-provider.md#prove-commit) ### `submit_windowed_post` -A storage provider needs to periodically submit a [Proof-of-Spacetime](../glossary.md#proofs) to prove that they are still storing the data they promised. Multiple proofs can be submitted at once. +A storage provider needs to periodically submit a [Proof-of-Spacetime](../../glossary.md#proofs) to prove that they are still storing the data they promised. Multiple proofs can be submitted at once. | Name | Description | Type | | ------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------- | @@ -172,7 +173,7 @@ Where `submit-windowed-post.json` is a file with contents similar to: } ``` -[^submit_windowed_post]: Read more about the `submit-windowed-post` command in [_Storagext CLI/Subcommand `storage-provider`/`submit-windowed-post`_](../storagext-cli/storage-provider.md#submit-windowed-post) +[^submit_windowed_post]: Read more about the `submit-windowed-post` command in [_Storagext CLI/Subcommand `storage-provider`/`submit-windowed-post`_](../../storagext-cli/storage-provider.md#submit-windowed-post) ### `declare_faults` @@ -215,7 +216,7 @@ Where `fault-declaration.json` is a file with contents similar to: ] ``` -[^declare_faults]: Read more about the `declare-faults` command in [_Storagext CLI/Subcommand `storage-provider`/`declare-faults`_](../storagext-cli/storage-provider.md#declare-faults) +[^declare_faults]: Read more about the `declare-faults` command in [_Storagext CLI/Subcommand `storage-provider`/`declare-faults`_](../../storagext-cli/storage-provider.md#declare-faults) ### `declare_faults_recovered` @@ -257,7 +258,47 @@ Where `fault-declaration.json` is a file with contents similar to: ] ``` -[^declare_faults_recovered]: Read more about the `declare-faults-recovered` command in [_Storagext CLI/Subcommand `storage-provider`/`declare-faults-recovered`_](../storagext-cli/storage-provider.md#declare-faults-recovered) +[^declare_faults_recovered]: Read more about the `declare-faults-recovered` command in [_Storagext CLI/Subcommand `storage-provider`/`declare-faults-recovered`_](../../storagext-cli/storage-provider.md#declare-faults-recovered) + +### `terminate_sectors` + +A storage provider can terminate sectors with the `terminate_sectors` extrinsic. This requires the storage provider to have no unproven sectors. +`terminate_sectors` can process multiple terminations in a single extrinsic. + +| Name | Description | Type | +| -------------- | --------------------------------------- | ------------------------------------- | +| `terminations` | The sectors and partitions to terminate | An array of termination declarations. | + +Where the termination declarations contain: + +| Name | Description | Type | +| ----------- | -------------------------------------------------------------------------- | ------------------------ | +| `deadline` | The deadline the termination is targeting | Positive integer | +| `partition` | Partition index within the deadline containing the sector to be terminated | Positive integer | +| `sectors` | Sectors in the partition being terminated | Set of positive integers | + +#### Example + +Storage provider `//Alice` terminating sectors[^terminate_sectors] on deadline 0, partition 0, sector 1. + + +```bash +storagext-cli --sr25519-key "//Alice" storage-provider terminate-sectors @terminate-sectors.json +``` + +Where `terminate-sectors.json` is a file with contents similar to: + +```json +[ + { + "deadline": 0, + "partition": 0, + "sectors": [1] + } +] +``` + +[^terminate_sectors]: Read more about the `terminate-sectors` command in [_Storagext CLI/Subcommand `storage-provider`/`terminate-sectors`_](../../storagext-cli/storage-provider.md#terminate-sectors) ## Events @@ -270,12 +311,21 @@ The Storage Provider Pallet emits the following events: - `window_post_proof_type` - The proof type used by the storage provider for sealing sectors. - `sector_size` - Amount of space in each sector committed to the network by the storage provider. - `window_post_partition_sectors` - The number of sectors in each Window PoSt partition (proof). -- `SectorPreCommitted` - A storage provider has pre-committed some new sector after publishing some new deal. +- `SectorsPreCommitted` - A storage provider has pre-committed some new sectors after publishing some new deal. - `owner` - SS58 address of the storage provider. - - `sector` - The sector number being pre-committed. -- `SectorProven` - A storage provider has proven a sector that they previously pre-committed. + - `sectors` - An array with information about the sectors that are pre-committed. This information includes: + - `seal_proof` - The seal proof type used. + - `sector_number` - The sector number that is pre-committed. + - `sealed_cid` - Commitment of replication. + - `deal_ids` - Deal IDs that are activated during pre-commit. + - `expiration` - Expiration of the pre-committed sector. + - `unsealed_cid` - Commitment of data. +- `SectorsProven` - A storage provider has proven a sectors that they previously pre-committed. - `owner` - SS58 address of the storage provider. - - `sector_number` - The sector number that was proven. + - `sectors` - An array with information about the sectors that are proven. This information includes: + - `sector_number` - The sector number that is proven. + - `partition_number` - The partition number the proven sector is in. + - `deadline_idx` - The deadline index assigned to the proven sector. - `SectorSlashed` - A previously pre-committed sector, but not proven, has been slashed by the system because it has expired. - `owner` - SS58 address of the storage provider. - `sector_number` - The sector number that has been slashed because of expiry. @@ -297,6 +347,12 @@ The Storage Provider Pallet emits the following events: - `owner` - SS58 address of the storage provider. - `partition` - Partition number for which the PoSt was missed. - `sectors` - The sectors in the partition declared faulty by the system. +- `SectorsTerminated` - A storage provider has terminated some sectors. + - `owner` - SS58 address of the storage provider. + - `terminations` - An array with information about the terminated sectors. This information includes: + - `deadline` - The deadline to which the terminated sectors were assigned. + - `partition` - The partition number within the deadline containing the terminated sectors. + - `sectors` - The sectors in the partition that have been terminated. ## Errors @@ -315,7 +371,7 @@ The Storage Provider Pallet actions can fail with the following errors: - `SectorNumberAlreadyUsed` - A storage provider tries to pre-commit a sector number that has already been used. - `ExpirationBeforeActivation` - A storage provider tries to pre-commit a sector where that sector expires before activation. - `ExpirationTooSoon` - A storage provider tries to pre-commit a sector with a total lifetime less than MinSectorExpiration. -- `ExpirationTooLong` - A storage provider tries to pre-commit a sector with an expiration that exceeds `MaxSectorExpirationExtension`. +- `ExpirationTooLong` - A storage provider tries to pre-commit a sector with an expiration that exceeds `MaxSectorExpiration`. - `MaxSectorLifetimeExceeded` - A storage provider tries to pre-commit a sector with a total lifetime that exceeds `SectorMaximumLifetime`. - `InvalidCid` - Emitted when a storage provider submits an invalid unsealed CID when trying to pre-commit a sector. - `ProveCommitAfterDeadline` - A storage provider has tried to prove a previously pre-committed sector after the proving deadline. @@ -323,31 +379,54 @@ The Storage Provider Pallet actions can fail with the following errors: - `InvalidUnsealedCidForSector` - This error is emitted when the declared unsealed_cid for pre_commit is different from the one calculated by the system. - `FaultDeclarationTooLate` - A fault declaration was submitted after the fault declaration cutoff. The fault declaration can be submitted after the upcoming deadline is closed. - `FaultRecoveryTooLate` - A fault recovery was submitted after the fault recovery cutoff. The fault recovery can be submitted after the upcoming deadline is closed. -- `DeadlineError` - An error was encountered in the deadline module. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. -- `PartitionError` - An error was encountered in the partition module. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. -- `StorageProviderError` - An error was encountered in the storage provider module. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. -- `SectorMapError` - An error was encountered in the sector map module. These errors can be: - - `FailedToInsertSector` - Internal bounds violation with Sectors. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. - - `FailedToInsertPartition` - Internal bounds violation with partitions. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. -- `CouldNotActivateSector` - Failure during prove commit when trying to convert a previously pre-committed sector due to a programming error. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. -- `CouldNotVerifySectorForPreCommit` - Failure during pre-commit due to the [commd](../glossary.md#commitment-of-data) calculation failing due to a programming error. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. +- `CouldNotTerminateDeals` - Emitted when trying to terminate sector deals fails. +- `InvalidDeadlineSubmission` - Emitted when an error occurs when submitting PoSt. +- `CouldNotVerifySectorForPreCommit` - Failure during pre-commit due to the [commd](../../glossary.md#commitment-of-data) calculation failing due to a programming error. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. - `SlashingFailed` - Slashing of funds fails due to a programmer error. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. - `ConversionError` - Due to a programmer error. Please [report an issue](https://github.com/eigerco/polka-storage-book/issues/new) to the developers. +- `GeneralPalletError` - An error ocurred in on of the pallet modules. These errors can be: + - `PartitionErrorFailedToAddSector` - Emitted when adding sectors fails. + - `PartitionErrorDuplicateSectorNumber` - Emitted when trying to add a sector number that has already been used in this partition. + - `PartitionErrorFailedToAddFaults` - Emitted when adding in the partition faults fails. + - `PartitionErrorSectorsNotLive` - Emitted when trying to remove sectors that are not live. + - `PartitionErrorFailedToRemoveRecoveries` - Emitted when removing recovering sectors from the partition fails. + - `PartitionErrorUnexpectedRecoveries` - Emitted when encountering unexpected recoveries while popping expired sectors. + - `PartitionErrorExpiredSectorsAlreadyTerminated` - Emitted when trying to pop expired sectors that are already terminated. + - `DeadlineErrorDeadlineIndexOutOfRange` - Emitted when the passed in deadline index supplied for `submit_windowed_post` is out of range. + - `DeadlineErrorDeadlineNotFound` - Emitted when a trying to get a deadline index but fails because that index does not exist. + - `DeadlineErrorCouldNotConstructDeadlineInfo` - Emitted when constructing `DeadlineInfo` fails. + - `DeadlineErrorPartitionAlreadyProven` - Emitted when a proof is submitted for a partition that is already proven. + - `DeadlineErrorPartitionNotFound` - Emitted when trying to retrieve a partition that does not exit. + - `DeadlineErrorProofUpdateFailed` - Emitted when trying to update proven partitions fails. + - `DeadlineErrorMaxPartitionsReached` - Emitted when max partition for a given deadline have been reached. + - `DeadlineErrorCouldNotAddSectors` - Emitted when trying to add sectors to a deadline fails. + - `DeadlineErrorSectorsNotFound` - Emitted when trying to use sectors which haven't been prove committed yet. + - `DeadlineErrorSectorsNotFaulty` - Emitted when trying to recover non-faulty sectors. + - `DeadlineErrorCouldNotAssignSectorsToDeadlines` - Emitted when assigning sectors to deadlines fails. + - `DeadlineErrorFailedToUpdateFaultExpiration` - Emitted when trying to update fault expirations fails. + - `StorageProviderErrorMaxPreCommittedSectorExceeded` - Happens when an SP tries to pre-commit more sectors than SECTOR_MAX. + - `StorageProviderErrorSectorNotFound` - Happens when trying to access a sector that does not exist. + - `StorageProviderErrorSectorNumberInUse` - Happens when a sector number is already in use. + - `SectorMapErrorFailedToInsertSector` - Emitted when trying to insert sector(s) fails. + - `SectorMapErrorFailedToInsertPartition` - Emitted when trying to insert partition fails. + - `ExpirationQueueErrorExpirationSetNotFound` - Expiration set not found. + - `ExpirationQueueErrorSectorNotFound` - Sector not found in expiration set. + - `ExpirationQueueErrorInsertionFailed` - Insertion into the expiration queue failed. ## Pallet constants The Storage Provider Pallet has the following constants: -| Name | Description | Value | -| ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | -| `WPoStProvingPeriod` | The average [period](../glossary.md#proving-period) for proving all sectors maintained by a storage provider. | 4 Minutes | -| `WPoStChallengeWindow` | The period immediately before a deadline during which a challenge can be generated by the chain and the requisite proofs computed. | 2 Minutes | -| `WPoStChallengeLookBack` | This period allows the storage providers to start working on the [PoSt](../glossary.md#post) before the deadline is officially opened to receiving a [PoSt](../glossary.md#post). | 1 Minute | -| `WPoStPeriodDeadlines` | Represents how many challenge deadlines there are in one proving period. Closely tied to `WPoStChallengeWindow`. | 48 | -| `MinSectorExpiration` | Minimum time past the current block a sector may be set to expire. | 5 Minutes | -| `MaxSectorExpirationExtension` | Maximum time past the current block a sector may be set to expire. | 60 Minutes | -| `SectorMaximumLifetime` | Maximum time a sector can stay in pre-committed state. | 120 Minutes | -| `MaxProveCommitDuration` | Maximum time between [pre-commit](#pre_commit_sector) and [proving](#prove_commit_sector) the committed sector. | 5 Minutes | -| `MaxPartitionsPerDeadline` | Maximum number of partitions that can be assigned to a single deadline. | 3000 | -| `FaultMaxAge` | Maximum time a [fault](../glossary.md#fault) can exist before being removed by the pallet. | 210 Minutes | -| FaultDeclarationCutoff | Time before a deadline opens that a storage provider can declare or recover a fault. | 2 Minutes | +| Name | Description | Value | +| ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- | +| `WPoStProvingPeriod` | The average [period](../../glossary.md#proving-period) for proving all sectors maintained by a storage provider. | 4 Minutes (40 Blocks) | +| `WPoStChallengeWindow` | The period immediately before a deadline during which a challenge can be generated by the chain and the requisite proofs computed. | 2 Minutes (20 Blocks) | +| `WPoStChallengeLookBack` | This period allows the storage providers to start working on the [PoSt](../../glossary.md#post) before the deadline is officially opened to receiving a [PoSt](../../glossary.md#post). | 1 Minute (10 Blocks) | +| `WPoStPeriodDeadlines` | Represents how many challenge deadlines there are in one proving period. Closely tied to `WPoStChallengeWindow`. | 48 | +| `MinSectorExpiration` | Minimum time past the current block a sector may be set to expire. | 5 Minutes (50 Blocks) | +| `MaxSectorExpiration` | Maximum time past the current block a sector may be set to expire. | 60 Minutes (600 Blocks) | +| `SectorMaximumLifetime` | Maximum time a sector can stay in pre-committed state. | 120 Minutes (1200 Blocks) | +| `MaxProveCommitDuration` | Maximum time between [pre-commit](#pre_commit_sectors) and [proving](#prove_commit_sectors) the committed sector. | 5 Minutes (50 Blocks) | +| `MaxPartitionsPerDeadline` | Maximum number of partitions that can be assigned to a single deadline. | 3000 | +| `FaultMaxAge` | Maximum time a [fault](../../glossary.md#fault) can exist before being removed by the pallet. | 210 Minutes (2100 Blocks) | +| FaultDeclarationCutoff | Time before a deadline opens that a storage provider can declare or recover a fault. | 2 Minutes (20 Blocks) | diff --git a/src/architecture/polka-storage-provider-server.md b/src/architecture/polka-storage-provider-server.md new file mode 100644 index 0000000..84746a3 --- /dev/null +++ b/src/architecture/polka-storage-provider-server.md @@ -0,0 +1,119 @@ +# Polka Storage Provider — Server Architecture + + +The server has two main fronts, the JSON-RPC API which provides an interface for users to submit deal proposals to the storage provider, +and the HTTP API which consists of a single endpoint where users are to submit their data — as illustrated below. + +

+ +The user is first required to propose a deal, which once accepted by the storage provider (signaled by the return of a CID) allows the user to submit a file +(using `curl` for example) to the server; finally the user can then publish a signed deal to the storage provider. For more details, see the [File Upload Demo](../getting-started/demo-file-store.md). + +The responsibility then falls on the storage provider to seal, publish and activate the deal on the Polka Storage parachain. + +## JSON-RPC API + +The JSON-RPC endpoint exposes the following methods: + +
+info — which returns information about the Storage Provider. +

JSON-RPC Example

+
+{
+  "jsonrpc": "2.0",
+  "id": 0,
+  "method": "v0_info",
+  "params": []
+}
+
+
+ +
+propose_deal — accepts a deal proposal and returns a CID for the file upload. +

JSON-RPC Example

+
+{
+  "jsonrpc": "2.0",
+  "id": 0,
+  "method": "v0_propose_deal",
+  "params": [
+    {
+      "piece_cid": "baga6ea4seaqj527iqfb2kqhy3tmpydzroiigyaie6g3txai2kc3ooyl7kgpeipi",
+      "piece_size": 2048,
+      "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
+      "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y",
+      "label": "",
+      "start_block": 200,
+      "end_block": 250,
+      "storage_price_per_block": 500,
+      "provider_collateral": 1250,
+      "state": "Published"
+    }
+  ]
+}
+
+
+ +
+publish_deal — after a file has been uploaded, accepts a signed deal for publishing. +
+{
+  "jsonrpc": "2.0",
+  "id": 0,
+  "method": "v0_publish_deal",
+  "params": [
+    {
+      "deal_proposal": {
+        "piece_cid": "baga6ea4seaqj527iqfb2kqhy3tmpydzroiigyaie6g3txai2kc3ooyl7kgpeipi",
+        "piece_size": 2048,
+        "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
+        "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y",
+        "label": "",
+        "start_block": 100000,
+        "end_block": 100050,
+        "storage_price_per_block": 500,
+        "provider_collateral": 1250,
+        "state": "Published"
+      },
+      "client_signature": {
+        "Sr25519": "c835a1c5215fc017067d30a8f49df0c643233881e57d8bd7232f695e1d28c748e8872b45712dcb403e28792cd1fb2b6161053b3344d4f6664bafec77349abd80"
+      }
+    }
+  ]
+}
+
+
+ +## HTTP API + +The HTTP API exposes a single PUT method — `/upload/` where `` is the CID returned as a result of `propose_deal`. + + +## Sealing Pipeline + +As shown in the previous illustration, the sealing pipeline is responsible for gathering pieces into sectors, sealing said sectors and proving their storage. +To achieve that, the pipeline is (currently) composed of 3 main stages. + +![The sealine pipeline, currently composed of the stages: Add Piece, Pre Commit and Prove Commit](../images/architecture/storage-provider-pipeline.png) + +### Add Piece + +The Add Piece stage gathers pieces into unsealed sectors, preparing them for the next steps. + +Given we're currently only supporting sectors with 2KiB size, we're converting single pieces into sectors — +when a piece comes in, we convert it to a single sector, without gathering multiple pieces. + +### Pre Commit + +By itself, the Pre Commit has two inner stages — Pre Commit 1 (PC1) and Pre Commit 2 (PC2). +PC1 is responsible for generating the [Stacked Depth Robust Graph](https://spec.filecoin.io/algorithms/sdr/#section-algorithms.sdr.stacked-depth-robust-graphs), +while PC2 is responsible for handling the construction of the [Merkle tree and proofs](https://spec.filecoin.io/algorithms/sdr/#section-algorithms.sdr.merkle-proofs). +After this process is completed, the Pre Commit information is submitted to the chain for verification. + +### Prove Commit + +The Prove Commit stage is where the [Proof of Replication](https://docs.filecoin.io/basics/the-blockchain/proofs#proof-of-replication-porep) is generated, +after generation it is submitted to the network for validation and the sector is finally marked as `Active`, +signaling that the Storage Provider has effectively stored the sector and is ready to start performing regular proof submissions. + + diff --git a/src/background.md b/src/background.md new file mode 100644 index 0000000..e31c213 --- /dev/null +++ b/src/background.md @@ -0,0 +1,93 @@ +# Background + +This page provides a quick background behind our approach to the storage challenge. + +## Why another storage system? + +In short, to be fully native to Polkadot (and as a bonus — no trusted execution environment is required). + +For a longer explanation, our vision is fully embedded in the Polkadot vision of the ubiquitous supercomputer; and as you know, a computer requires storage. + +Achieving storage native to Polkadot means using DOT as *the* token for our solution, instead of creating a parallel economy relying on yet another token, further fragmenting the space. + +## Why not X? + +We've decided to build our storage solution based on the ideas behind Filecoin, not because it's different from the other available networks in Polkadot, but rather because Filecoin does a lot of things right, and we want to bring them to Polkadot. + +For those unfamiliar with Filecoin, it is a blockchain network that provides a *file* storage marketplace. In a nutshell, people provide storage space (a storage provider), creating several offers (distinguished by price, reputation, etc) and clients can pick one of these storage providers to store their data for them. + +This raises the question — *How does the network know that the storage provider has my data?* + +Therein lies the crux of Filecoin! You can solve this in multiple ways, each with different levels of flexibility, but we will outline two. + +### Merkle Trees and Proofs + +> If you’re not familiar with Merkle trees, you can read the explainer from BitPanda — +> + +Consider that you build a Merkle tree out of a file, the root of the Merkle tree is derived of several layers of hashes meaning that if you change one of those layers, such as a leaf, the resulting tree will have a different root — as illustrated below, `D3` was changed to `D7` which cascades into a different final hash. + +Two Merkle trees. + On the left, a Merkle tree with colored nodes; + in grey, data sectors the tree is built on, + in yellow, hashes calculated from the sectors and previous hashes. + On the right, a Merkle tree with colored nodes; + in grey, data sectors the tree is built on, + in darker grey a different sector from the first tree (D3 became D7), + in yellow, hashes calculated from the sectors and previous hashes, + in red, the hashes calculated from the sectors and previous hashes showing the path affected by the change. + +Merkle proofs are similar, but instead of sharing the whole file, the tree alone can be shared, if the verifier and the provider have different trees, it means somewhere down the tree there’s a difference! + +With this in mind, you can challenge storage providers holding a given file by selecting a random data leaf (i.e. a piece of the file) and sending them a random number — as illustrated in the figure below, the random number `R` is concatenated to the sector `D3` — the idea is, the storage provider cannot guess the random number, so, if they’re able to generate a tree with the random number, it must mean they have the file! + +The view of the Merkle tree on the storage provider side, the tree has nodes with multiple colors. + In red, the nodes that are modified by D3 concatenated with R. + In yellow, the calculated hashes from the sectors and previous hashes. + In grey, the data blocks the tree is built on. + +At the same time you, the challenger, must also build a Merkle tree on your end, you do this by receiving the random data leaf that you selected and building the tree, in the end, if they do not match, the storage provider is cheating you! + +This is illustrated in the following picture, you — the verifier — just received the leaf `D3` and the random number `R`, you concatenate them together (`D3||R`) and you’re ready to recompute the tree. In red, you find the nodes directly affected by the change, these are the ones that **must** be recomputed — of course, you can always compute the tree from scratch; using `h(D1), h(D2), ...` but that is wasteful as the nodes marked in blue and their children did not change, and as such, you can just reuse them. + +The view of the Merkle tree on the verifier side, the tree has nodes with multiple colors. + In red, the nodes that are modified by D3 concatenated with R. + In blue, the nodes that are absolutely require to compute the modified tree. + In white, data blocks that are not necessary for the computation. + In yellow, the hash of the white nodes, not strictly necessary for the computation. + +This approach is great for small batches of data, it does not require any special hardware and can be implemented with different kinds of hashes, providing flexibility and lowering entry requirements for storage providers. + +However, this approach does not scale well because the challenger is required to receive both the Merkle tree *and* the random data leaf that is being challenged — as illustrated below. + +Challenge protocol + +Now, consider that you need to do this over and over while you keep the file, over time, all those data blocks start accumulating and you end up transferring a lot of data over the network! Furthermore, you can't just request any size of data; if it is too small, the storage provider may cheat and brute force a solution; if it is too big, the transfer may take too long to be practical. + +### Zero Knowledge Proofs + +So, we've established that we can't transfer much data through the network, but it cannot be possible to brute force a solution. That's where Filecoin's solution comes in, they had the brilliant insight that you can use the random challenges along with zero-knowledge proofs for that. Filecoin's zero-knowledge proofs are constant in size, easy to verify and hard to fake. + +The generation of these proofs do not require special hardware features like trusted platform modules — you can generate a proof using your CPU, however, you will need a GPU if you want to generate proofs for larger files in practical time. + +In Filecoin, uploaded files (or deals) are combined into sectors, which the zero knowledge proofs are based on (and [directed acyclic graphs](https://www.youtube.com/watch?v=8_9ONpyRZEI), but we’re not covering that here). At this point Merkle trees are built over the original file and the final sector, both in its unsealed and sealed state. The root of each tree is then used when constructing the replication proof, similarly, the root of the sealed sector is used for the proofs of storage over time. + +Proof validation requires fewer resources than the generation step, enabling us to verify the storage proofs inside the Polkadot runtime. Their small size translates to less stress on the network; for example, Filecoin proofs can go from 192 bytes to a few KB in size, in comparison, a 1080p video frame, encoded using H.264 will be between 100 and 500 KB — note that video will usually be streamed at 24 frames per second or higher, the proof size pales in comparison! + +#### Relevant Links + +* +* diff --git a/src/getting-started/building/docker.md b/src/getting-started/building/docker.md new file mode 100644 index 0000000..b3e750e --- /dev/null +++ b/src/getting-started/building/docker.md @@ -0,0 +1,78 @@ +# Docker Setup + +This guide will outline how to setup your environment using Docker to get started with the Polka Storage parachain. + +## Pre-requisites + +Install Docker on your system by following the [Docker install instructions](https://docs.docker.com/engine/install/). + +> Using Podman instead of Docker **may** work, however, we **do not support** Podman! + +## Dockerfile setup + +All docker builds are composed of 4 stages. + +1. Set up [`cargo chef`](https://github.com/LukeMathWalker/cargo-chef), this caches the Rust dependencies for faster builds. +2. Planning — `cargo chef` analyzes the current project to determine the minimum subset of file required to build it an cache the dependencies. +3. Build — `cargo chef` checks the project skeleton identified in the planner stage and builds it to cache dependencies. +4. Runtime — sets up the runtime with Debian and imports the binary build in the previous stage. + +## Building & Running + +Clone the repository and go into the directory: + +```shell +git clone git@github.com:eigerco/polka-storage.git +cd polka-storage +``` + +You can find Dockerfiles for each binary under the `docker/` folder. +To build the images manually you can use the following command: + +```bash +docker build \ + --build-arg VCS_REF="$(git rev-parse HEAD)" \ + --build-arg BUILD_DATE="$(date -u +'%Y-%m-%dT%H:%M:%SZ')" \ + -t :"$(cargo metadata --format-version=1 --no-deps | jq -r '.packages[0].version')" \ + --file ./docker/dockerfiles/.Dockerfile \ + . +``` + +Where you can replace `` by one of the following: + +- `polka-storage-node` +- `polka-storage-provider-server` +- `polka-storage-provider-client` +- `mater-cli` +- `storagext-cli` + +To run the images manually, you apply the same pattern to the following command: + +```bash +docker run -it polkadotstorage.azurecr.io/:"$(cargo metadata --format-version=1 --no-deps | jq -r '.packages[0].version')" +``` + +### Just recipes + +To simplify the building process, we've written some [Just](https://github.com/casey/just) recipes. + +#### Build recipes + +| Command | Description | +| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | +| `build-mater-docker` | Builds the `mater` CLI image which is used by storage clients to convert files to CARv2 format and extract CARv2 content. | +| `build-polka-storage-node-docker` | Builds the Polka Storage parachain node image. | +| `build-polka-storage-provider-server-docker` | Builds the Storage Provider server image. | +| `build-polka-storage-provider-client-docker` | Builds the Storage Provider client image. | +| `build-storagext-docker` | Builds the `storagext` CLI image used to execute extrinsics. | +| `build-docker-all` | Builds all the images above, this might take a while to complete. | + +#### Running recipes + +| Command | Description | +| ------------------------------------------ | ------------------------------------------------------------------------------------------ | +| `run-mater-docker` | Runs the image, opening a shell with access to the `mater-cli` binary. | +| `run-polka-storage-node-docker` | Runs the `polka-storage-node` inside the built Docker image. | +| `run-polka-storage-provider-server-docker` | Runs the image, opening a shell with access to the `polka-storage-provider-server` binary. | +| `run-polka-storage-provider-client-docker` | Runs the image, opening a shell with access to the `polka-storage-provider-client` binary. | +| `run-storagext-docker` | Runs the image, opening a shell with access to the `storagext-cli` binary. | diff --git a/src/getting-started/building/index.md b/src/getting-started/building/index.md new file mode 100644 index 0000000..0313301 --- /dev/null +++ b/src/getting-started/building/index.md @@ -0,0 +1,26 @@ +# Building + +The following chapters will cover how to build the Polka Storage parachain using multiple methods. + +
+Quick reminder that Windows is not part of the supported operating systems. +As such, the following guides have not been tested on Windows. +
+ + +We'll be building 5 artifacts: + +* `polka-storage-node` — the Polka Storage Polkadot parachain node. +* `polka-storage-provider-server` — the Polka Storage Storage Provider, + responsible for accepting deals, storing files and executing storage proofs. +* `polka-storage-provider-client` — the Polka Storage Storage Provider client, + this CLI tool has several utilities to interact with the Storage Provider server, + such as proposing and publishing deals, as well as some wallet utilities and proof demos. +* `mater-cli` — the Mater CLI enables you to convert files into CARv2 archives, + an essential part of preparing files for submission to the network. +* `storagext-cli` — the Storagext CLI is a lower-level tool to manually run the Polka Storage extrinsics. + +To build these artifacts, we provide two main methods: + +* [Building from source](./source.md) +* [Building with Docker](./docker.md) diff --git a/src/getting-started/building/source.md b/src/getting-started/building/source.md new file mode 100644 index 0000000..59126ed --- /dev/null +++ b/src/getting-started/building/source.md @@ -0,0 +1,129 @@ +# Building from source + +This guide will outline how to setup your environment to build the Polka Storage parachain, +we cover how to build the binaries directly on your system or using [Nix](https://nixos.org/download/) to ease the process. + +* [Get the code](#get-the-code) +* [System dependencies](#system-dependencies) +* [Using Nix](#using-nix) + * [Pre-requisites](#pre-requisites) +* [Building](#building) + +## Get the code + +To get started, first clone the repository and enter the repository's directory: + +```bash +git clone git@github.com:eigerco/polka-storage.git +cd polka-storage +``` + + +## System dependencies + +To build the binaries directly on your system you will need the following tools: + +* Rust 1.77 — you can install it using [`rustup`](https://rustup.rs/) and its [guide](https://rust-lang.github.io/rustup/installation/other.html) for help. +* Other dependencies — keep reading, we'll get to it after the end of this list! +* `just` (optional) — (after installing Rust) you can use `cargo install just` or check the [official list of packages](https://just.systems/man/en/packages.html). + +The dependencies mentioned are for Linux distros using the `apt` family of package managers. +Different systems may use different package managers, as such, they may require you to find the equivalent package. + +To install the required dependencies run the following commands: + +```shell +$ sudo apt update +$ sudo apt install -y libhwloc-dev \ + opencl-headers \ + ocl-icd-opencl-dev \ + protobuf-compiler \ + clang \ + build-essential \ + git \ + curl +``` + +## Using Nix + +You can use Nix to simplify the building process, +if you're just taking the network for test-drive this is a great method to get started. + +Nix will take care of setting up all the dependencies for you! +If you're curious, you can read more about using Nix in [fasterthanlime's blog](https://fasterthanli.me/series/building-a-rust-service-with-nix/part-9), +the [official Nix guide](https://nixos.org/learn/) or [Determinate Systems' Zero to Nix guide](https://zero-to-nix.com/). + +
+Binaries built using Nix will not work on other systems since they will be linked with Nix specific paths. +
+ +### Pre-requisites + +- `nix` — which you can install by following the [official guide](https://nixos.org/download/) + or using the [Determinate Systems installer](https://github.com/DeterminateSystems/nix-installer) — the latter being usually more reliable on MacOS systems. +- `direnv` (optional) — which you can install by following the [official guide](https://direnv.net/docs/installation.html) + +If you're using `direnv`, when going into the cloned directory for the first time `nix` will activate automatically and +install the required packages, this make take some time. + +If you're _not_ using `direnv`, you will need to run `nix develop` to achieve the same effect — +for more information refer to the official Nix guide — https://nix.dev/manual/nix/2.17/command-ref/new-cli/nix3-develop. + +## Building + +After all this setup, it is time to start building the binaries, which you can do manually using the following command: + +
+ +When building `polka-storage-node` you should add `--features polka-storage-runtime/testnet` which enables the testnet configuration; all the code in the repo is currently targeting this feature, not including it may lead to unexpected behavior. + +When building `storagext-cli` you may want to add `--features storagext/insecure_url` which enables using non-TLS HTTP and WebSockets. +
+ +```bash +cargo build --release -p +``` + +Where `` is one of: + +- `polka-storage-node` +- `polka-storage-provider-client` +- `polka-storage-provider-server` +- `storagext-cli` +- `mater-cli` + + +For more information on what each binary does, refer to [Building](./index.md). + +### Just recipes + +To simplify the building process, we've written some [Just](https://github.com/casey/just) recipes. + +| Command | Description | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | +| `build-polka-storage-node` | Builds the Polka Storage parachain node. | +| `build-polka-storage-provider-server` | Builds the Storage Provider server binary. | +| `build-polka-storage-provider-client` | Builds the Storage Provider client binary. | +| `build-storagext-cli` | Builds the `storagext` CLI used to execute extrinsics. | +| `build-mater-cli` | Builds the `mater` CLI which is used by storage clients to convert files to CARv2 format and extract CARv2 content. | +| `build-binaries-all` | Builds all the binaries above, this may take a while (but at least `cargo` reuses artifacts). | + +## Running + +After building the desired binaries, you can find them under the `target/release` folder +(or `target/debug` if you didn't use the `-r`/`--release` flag). + +Assuming you're in the project root, you can run them with the following command: + +```bash +$ target/release/ +``` + +Where `` is one of: +* `polka-storage-node` +* `polka-storage-provider-server` +* `polka-storage-provider-client` +* `mater-cli` +* `storagext-cli` + +> Additionally, you can move them to a folder under your `$PATH` and run them as you would with any other binary. diff --git a/src/getting-started/car-server.md b/src/getting-started/car-server.md deleted file mode 100644 index 4281e0a..0000000 --- a/src/getting-started/car-server.md +++ /dev/null @@ -1,60 +0,0 @@ -# CAR server - -It is an HTTP server that converts arbitrary content into a [CARv2](https://ipld.io/specs/transport/car/carv2/) file and serves it over HTTP - supporting the latest CARv2 format, -which is not yet entirely supported by other crates in the Rust ecosystem. -The next steps describe how to run the server locally and use it to upload and download files. - -
-The server is a proof of concept, showcasing our CARv2 implementation, but it is not intended to be used in production. -Anyone can upload and download files without authentication or authorization. -
- -## Start the server - -1. Create a Docker volume to store uploaded files: - -`docker volume create storage_provider` - -2. Start the server: - -``` -docker run \ - -p 127.0.0.1:9000:9000 \ - --mount source=storage_provider,destination=/app/uploads \ - polkadotstorage.azurecr.io/polka-storage-provider:0.1.0 storage \ - --listen-addr 0.0.0.0:9000 -``` - -- `-p 127.0.0.1:9000:9000`: Maps port `9000` on the localhost to port `9000` on the container. -- `--mount source=storage_provider,destination=/app/uploads`: Mounts the `storage_provider` volume to `/app/uploads` inside the container. -- `polkadotstorage.azurecr.io/polka-storage-provider:0.1.0 storage`: Runs the `polkadotstorage.azurecr.io/polka-storage-provider:0.1.0` image with the `storage` command. -- `--listen-addr 0.0.0.0:9000`: Configures the server to listen on all available network interfaces. - -## Verifying the Setup - -After setting up and starting the CAR server, it's essential to verify that everything works correctly. -Follow these steps to ensure your setup works as expected: - -1. Upload a test file using the instructions in the [Upload a file](../storage-provider-cli/storage.md#upload-a-file) section. Make sure to note the CID returned by the server. - -2. Download the CAR file using the retrieved CID, following the steps in the [Download the CAR File](../storage-provider-cli/storage.md#download-the-car-file) section. - -3. [Optional] Verify the contents of the downloaded CAR file. Using, for example, [go-car](https://github.com/ipld/go-car/tree/master/cmd/car#install)'s `inspect` command: - - ```bash - car inspect - ``` - - The user can use `debug` for more detailed output: - - ```bash - car debug - ``` - - If the user desires, they can extract the contents of the file: - - ```bash - car extract - ``` - -If a file can be successfully uploaded, the server produces a CID, allows download the corresponding CAR file, and verify its contents, the CAR server setup is working correctly. diff --git a/src/getting-started/demo-file-prove.md b/src/getting-started/demo-file-prove.md new file mode 100644 index 0000000..900587c --- /dev/null +++ b/src/getting-started/demo-file-prove.md @@ -0,0 +1,56 @@ +# Proving a file + +To store the file according to the protocol, Storage Provider has to assign it to a sector, pre-commit and then prove it! +That's a lot of steps, but this is handled automatically, behind the scenes by the [pipeline](../architecture/polka-storage-provider-server.md#sealing-pipeline), then the file is eventually [published](../storage-provider-cli/client/index.md#publish-deal). + +Here are excerpts from Storage Provider Node after executing the [store a file scenario](./demo-file-store.md): + +```log +2024-11-11T12:34:21.430693Z INFO start_rpc_server: polka_storage_provider_server::rpc: Starting RPC server at 127.0.0.1:8000 +2024-11-11T12:34:21.430870Z INFO start_upload_server: polka_storage_provider_server::storage: Starting HTTP storage server at: 127.0.0.1:8001 +2024-11-11T12:34:21.431984Z INFO start_rpc_server: polka_storage_provider_server::rpc: RPC server started +2024-11-11T12:35:07.883255Z INFO request{method=PUT matched_path="/upload/:cid" request_id=e71d7e49-0272-435e-899e-a12a5d639268}:upload: polka_storage_provider_server::storage: CAR file created final_content_path="/var/folders/51/ch08ltd95bxcwpvskd28wr5h0000gp/T/Xvm5m7j/deals_storage/car/bafkreihoxd7eg2domoh2fxqae35t7ihbonyzcdzh5baevxzrzkaakevuvy.car" +2024-11-11T12:37:29.258216Z INFO add_piece: polka_storage_provider_server::pipeline: Adding a piece... +2024-11-11T12:37:29.258785Z INFO polka_storage_provider_server::pipeline: Preparing piece... +2024-11-11T12:37:29.259375Z INFO polka_storage_provider_server::pipeline: Adding piece... +2024-11-11T12:37:29.261621Z INFO add_piece: polka_storage_provider_server::pipeline: Finished adding a piece +2024-11-11T12:37:29.261979Z INFO polka_storage_provider_server::pipeline: Add Piece for piece Commitment { commitment: [...], kind: Piece }, deal id 0, finished successfully. +2024-11-11T12:37:29.262023Z INFO precommit: polka_storage_provider_server::pipeline: Starting pre-commit +2024-11-11T12:37:29.262258Z INFO precommit: polka_storage_provider_server::pipeline: Padded sector, commencing pre-commit and getting last finalized block +2024-11-11T12:37:29.263185Z INFO precommit: polka_storage_provider_server::pipeline: Current block: 35 +2024-11-11T12:37:29.263852Z INFO filecoin_proofs::api::seal: seal_pre_commit_phase1:start: SectorId(1) +2024-11-11T12:37:29.275251Z INFO storage_proofs_porep::stacked::vanilla::proof: replicate_phase1 +2024-11-11T12:37:29.275814Z INFO storage_proofs_porep::stacked::vanilla::graph: using parent_cache[64 / 64] +2024-11-11T12:37:29.276105Z INFO storage_proofs_porep::stacked::vanilla::cache: parent cache: opening /var/tmp/filecoin-parents/v28-sdr-parent-3f0eef38bb48af1f48ad65e14eb85b4ebfc167cec18cd81764f6d998836c9899.cache, verify enabled: false +2024-11-11T12:37:29.277633Z INFO storage_proofs_porep::stacked::vanilla::proof: single core replication +2024-11-11T12:37:29.277644Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: generate labels +2024-11-11T12:37:29.277681Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: generating layer: 1 +2024-11-11T12:37:29.277915Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: storing labels on disk +2024-11-11T12:37:29.278316Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: generated layer 1 store with id layer-1 +2024-11-11T12:37:29.278328Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: setting exp parents +2024-11-11T12:37:29.278336Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: generating layer: 2 +2024-11-11T12:37:29.278418Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: storing labels on disk +2024-11-11T12:37:29.278735Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: generated layer 2 store with id layer-2 +2024-11-11T12:37:29.278745Z INFO storage_proofs_porep::stacked::vanilla::create_label::single: setting exp parents +2024-11-11T12:37:29.278761Z INFO filecoin_proofs::api::seal: seal_pre_commit_phase1:finish: SectorId(1) +[...] +2024-11-11T12:37:29.313831Z INFO storage_proofs_core::data: dropping data /var/folders/51/ch08ltd95bxcwpvskd28wr5h0000gp/T/Xvm5m7j/deals_storage/sealed/1 +2024-11-11T12:37:29.314137Z INFO filecoin_proofs::api::seal: seal_pre_commit_phase2:finish +2024-11-11T12:37:29.314165Z INFO precommit: polka_storage_provider_server::pipeline: Created sector's replica: PreCommitOutput { } +[...] +2024-11-11T12:37:57.324204Z INFO precommit: polka_storage_provider_server::pipeline: Successfully pre-commited sectors on-chain: [SectorsPreCommitted { block: 39, [...] }] +2024-11-11T12:37:57.324292Z INFO polka_storage_provider_server::pipeline: Precommit for sector 1 finished successfully. +2024-11-11T12:37:57.324345Z INFO prove_commit: polka_storage_provider_server::pipeline: Starting prove commit +2024-11-11T12:37:57.325705Z INFO prove_commit: polka_storage_provider_server::pipeline: Wait for block 49 to get randomness +2024-11-11T12:39:05.518784Z INFO storage_proofs_porep::stacked::vanilla::proof: generating interactive vanilla proofs +2024-11-11T12:39:05.529259Z INFO bellperson::groth16::prover::native: Bellperson 0.26.0 is being used! +2024-11-11T12:39:06.634632Z INFO bellperson::groth16::prover::native: synthesis time: 1.105318708s +2024-11-11T12:39:06.634659Z INFO bellperson::groth16::prover::native: starting proof timer +[...] +2024-11-11T12:39:23.728566Z INFO bellperson::groth16::prover::native: prover time: 17.094277959s +2024-11-11T12:39:23.737186Z INFO prove_commit: polka_storage_provider_server::pipeline: Proven sector: 1 +``` + +After that, Storage Provider needs to continously [submit a PoSt](../architecture/pallets/storage-provider.md#submit_windowed_post) to prove that they are still storing the file. If they do not, they'll be slashed. +We have not yet integrated the logic for PoSt verification with Storage Provider node, but the logic on-chain has been implemented. + diff --git a/src/getting-started/demo-file-store.md b/src/getting-started/demo-file-store.md new file mode 100644 index 0000000..74871c6 --- /dev/null +++ b/src/getting-started/demo-file-store.md @@ -0,0 +1,149 @@ +# Storing a file + +
+Before reading this guide, please follow the local testnet guide and storage provider guide. +You should have a working testnet and a Storage Provider running! +
+ + +## Storage Client +The Polkadot logo + +Alice is a [Storage User](../glossary.md#storage-user) and wants to store an image of her lovely Polkadot logo [`polkadot.svg`](../images/polkadot.svg) in the Polka Storage [parachain](../glossary.md#parachain). + +Alice knows that she needs to prepare an image for storage and get its [CID](https://github.com/multiformats/cid). +To do so, she first converts it into a [CARv2 archive](https://ipld.io/specs/transport/car/carv2/) and gets the piece cid. + +```bash +$ mater-cli convert -q --overwrite polkadot.svg polkadot.car +bafkreihoxd7eg2domoh2fxqae35t7ihbonyzcdzh5baevxzrzkaakevuvy +$ polka-storage-provider-client proofs commp polkadot.car +{ + "cid": "baga6ea4seaqabpfwrqjcwrb4pxmo2d3dyrgj24kt4vqqqcbjoph4flpj2e5lyoq", + "size": 2048 +} +``` + +### Proposing a deal + +Afterwards, it's time to propose a deal, currently — i.e. while the network isn't live — +any deals will be accepted by Charlie (the Storage Provider). + + +Alice fills out the deal form according to a JSON template (`polka-logo-deal.json`): + +```json +{ + "piece_cid": "baga6ea4seaqabpfwrqjcwrb4pxmo2d3dyrgj24kt4vqqqcbjoph4flpj2e5lyoq", + "piece_size": 2048, + "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", + "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", + "label": "", + "start_block": 200, + "end_block": 250, + "storage_price_per_block": 500, + "provider_collateral": 1000100200, + "state": "Published" +} +``` + +* `piece_cid` — is the `cid` field from the previous step, where she calculated the piece commitment. It uniquely identifies the piece. +* `piece_size` — is the `size` field from the previous step, where she calculated the piece commitment. It is the size of the processed piece, not the original file! +* `client` — is the client's (i.e. the reader's) public key, encoded in bs58 format. + For more information on how to generate your own keypair, read the [Polka Storage Provider CLI/`client`/`wallet`](../storage-provider-cli/client/wallet.md). +* `provider` — is the storage provider's public key, encoded in bs58 format. + If you don't know your storage provider's public key, you can query it using `polka-storage-provider-client`'s `info` command. +* `label` — is an arbitrary string to be associated with the deal. +* `start_block` — is the deal's start block, it MUST be positive and lower than `end_block`. +* `end_block` — is the deal's end block, it must be positive and larger than `start_block`. +* `storage_price_per_block` — the storage price over the duration of a single block — e.g. if your deal is 20 blocks long, it will cost `20 * storage_price_per_block` in total. +* `provider_collateral` — the price to pay *by the storage provider* if they fail to uphold the deal. +* `state` — the deal state, only `Published` is accepted. + + +
+ +The `start_block` and `end_block` fields may need to be changed depending on the current block you are on. +The values `200` and `250` are solely for demonstration purposes and we encourage you to try other values! + +
+ +
+Variables subject to change depending on the chains state + +`start_block` - The start block **must** be after the current block. Check the polka storage node logs or use the polkadot.js UI for the current block and adjust the start_block value accordingly. + +`end_block` - The end block **must** be between 50 and 1800 blocks after `start_block`. + +See the [Storage Provider Constants](../architecture/pallets/storage-provider.md#pallet-constants) and the [Market Constant](../architecture/pallets/market.md#constants) for more information about the configuration variables + +
+ +When the deal is ready, she proposes it: + +```bash +$ polka-storage-provider-client propose-deal --rpc-server-url "http://localhost:8000" "@polka-logo-deal.json" +bagaaierab543mpropvi5mnmtptytnnlbr2j7vea7lowcugrqt7epanybw7ta +``` + +The storage provider replied with a CID — the CID of the deal Alice just sent — she needs to keep this CID for the next steps! + +Once the server has replied with the CID, she's ready to upload the file. +This can be done with just any tool that can upload a file over HTTP. +The server supports both [multipart forms](https://curl.se/docs/httpscripting.html#file-upload-post) and [`PUT`](https://curl.se/docs/httpscripting.html#put). + +```bash +$ curl --upload-file "polkadot.svg" "http://localhost:8001/upload/bagaaierab543mpropvi5mnmtptytnnlbr2j7vea7lowcugrqt7epanybw7ta" +baga6ea4seaqabpfwrqjcwrb4pxmo2d3dyrgj24kt4vqqqcbjoph4flpj2e5lyoq +``` + +### Publishing the deal + +Before Alice publishes a deal, she must ensure that she has the necessary funds available in the market escrow, to be able to pay for the deal: + +```bash +$ storagext-cli --sr25519-key "//Alice" market add-balance 25000000000 +[0x6489…a2c0] Balance Added: { account: 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY, amount: 25000000000 } +``` + +Finally, she can publish the deal by submitting her deal proposal along with your signature to the storage provider. + +To sign her deal proposal she runs: + +```bash +$ polka-storage-provider-client sign-deal --sr25519-key "//Alice" @polka-logo-deal.json +``` + +```json +{ + "deal_proposal": { + "piece_cid": "baga6ea4seaqabpfwrqjcwrb4pxmo2d3dyrgj24kt4vqqqcbjoph4flpj2e5lyoq", + "piece_size": 2048, + "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", + "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", + "label": "", + "start_block": 200, + "end_block": 250, + "storage_price_per_block": 500, + "provider_collateral": 1000100200, + "state": "Published" + }, + "client_signature": { + "Sr25519": "7eb8597441711984b7352bd4a118eac57341296724c20d98a76ff8d01ee64038f6a9881e492a98c3a190e7b600a8313d72e9f0edacb3e6df6b0b4507dabb9580" + } +} +``` + +> Hint: you can write the following command to *just* get the file: +> ``` +> polka-storage-provider-client sign-deal --sr25519-key "//Alice" @polka-logo-deal.json > signed-logo-deal.json +> ``` + +All that's left is to [publish the deal](../storage-provider-cli/client/index.md#publish-deal): + +```bash +$ polka-storage-provider-client publish-deal --rpc-server-url "http://localhost:8000" @signed-logo-deal.json +Successfully published deal of id: 0 +``` + +On Alice's side, that's it! diff --git a/src/getting-started/demo.md b/src/getting-started/demo.md deleted file mode 100644 index 91cfaf0..0000000 --- a/src/getting-started/demo.md +++ /dev/null @@ -1,263 +0,0 @@ -# Real-world use case demo - -
-Before reading this guide, please follow the local testnet guide and have a working testnet running! -
- -
- -For convenience's sake, we have a script that automates the actions in this guide. -The script is available at the following link: - - -Alternatively, download and run in a single step: - -```bash -wget https://polka-storage.s3.eu-central-1.amazonaws.com/demo.sh -chmod +x demo.sh -./demo.sh -``` - -
- -The script has some pre-requisites: -* The storagext-cli binary *must* exist and be present in the $PATH — - instructions on how to achieve that are available in the Local Testnet - Polka Storage Parachain chapter. -* You need to launch a *fresh* parachain, as the script needs to be run while the Charlie node is booting up; - the script will wait for the first block before starting. *This process may need more than one attempt.* - -
- -
- -A high-level overview with diagrams of the process described below can be found in Pallets section. - -## 1. Publishing a deal - -Charlie heard he could provide storage to people worldwide and earn some tokens, -so he decided to register as a [Storage Provider](../glossary.md). - -```bash -$ storagext-cli --sr25519-key "//Charlie" storage-provider register Charlie -[0xd14d…4b6a] Storage Provider Registered: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, info: Storage Provider Info: { peer_id: 3ZAB4sc5BS, window_post_proof_type: StackedDRGWindow2KiBV1P1, sector_size: _2KiB, window_post_partition_sectors: 2 }, proving_period_start: 21 } -``` - - - -Alice is a [Storage User](../glossary.md#storage-user) and wants to store an image of her lovely Husky (`husky.jpg`) in the Polka Storage [parachain](../glossary.md#parachain). - -Alice knows[^no-cid] that she needs to get a [CID](https://github.com/multiformats/cid) of the image, -so she [uploaded it to the CAR server](../storage-provider-cli/storage.md#upload-a-file) -and received the CID: `bafybeihxgc67fwhdoxo2klvmsetswdmwwz3brpwwl76qizbsl6ypro6vxq`. - -Alice heard somewhere[^no-sp-discovery] in the hallways of her favourite gym that Charlie is a Storage Provider. -She calls him (off-chain), and they negotiate a deal: - -`husky-deal.json` - -```json -[ - { - "piece_cid": "bafybeihxgc67fwhdoxo2klvmsetswdmwwz3brpwwl76qizbsl6ypro6vxq", - "piece_size": 1278, - "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", - "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", - "label": "My lovely Husky (husky.jpg)", - "start_block": 25, - "end_block": 50, - "storage_price_per_block": 1000000000, - "provider_collateral": 12500000000, - "state": "Published" - } -] -``` - -| Name | Value | Description | -| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | -| `piece_cid` | `bafybeihxgc67fwhdoxo2klvmsetswdmwwz3brpwwl76qizbsl6ypro6vxq` | The submitted file's CID | -| `piece_size` | `1278` | The submitted file's size | -| `client` | [`//Alice`](https://docs.substrate.io/reference/glossary/#dev-phrase) or `5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY` | The client's address | -| `provider` | [`//Charlie`](https://docs.substrate.io/reference/glossary/#dev-phrase) or `5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y` | The providers's address | -| `start_block` | `25` | When the deal should start | -| `end_block` | `50` | When the deal should end | -| `storage_price_per_block` | `1_000_000_000` | In [Plancks](../glossary.md#planck), the storage price per block (12 sec) | -| `provider_collateral` | `12_500_000_000` | In [Plancks](../glossary.md#planck), the price paid by the storage provider if they fail to uphold their end of the deal | - -After the negotiation, they need to [add funds to the Pallet](../pallets/market.md#add_balance) and then [publish their intent](../pallets/market.md#publish_storage_deals) so it can be checked by the parachain. -So here they go: - -```bash -# Adding balance to Alice's account -$ storagext-cli --sr25519-key "//Alice" market add-balance 25000000000 -[0xe0bd…06f9] Balance Added: { account: 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY, amount: 25000000000 } -# Adding balance to Charlie's account -$ storagext-cli --sr25519-key "//Charlie" market add-balance 12500000000 -[0x25aa…edd3] Balance Added: { account: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, amount: 12500000000 } -# Publishing their deal -$ storagext-cli --sr25519-key "//Charlie" market publish-storage-deals --client-sr25519-key "//Alice" "@husky-deal.json" -[0xd50b…dee6] Deal Published: { deal_id: 0, provider_account: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, client_account: 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY } -``` - -[^no-cid]: We have not provided a standalone command to generate CID out of the file. The CAR server is a temporary showcase component. -[^no-sp-discovery]: We have not yet implemented Storage Provider Discovery protocol. - -## 2. Committing a deal - -After the deals have been published, the rest is up to Charlie. -If Charlie does not behave appropriately, [pre-commit](../pallets/storage-provider.md#pre_commit_sector) and [prove](../pallets/storage-provider.md#prove_commit_sector) the deal by block 25 (`start_block`), -he will be slashed, and all his funds [(`provider_collateral`)](../glossary.md#collateral) will be gone.[^slash] -So he should do his part! - -`pre-commit-husky.json` - -```json -{ - "sector_number": 1, - "sealed_cid": "bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu", - "deal_ids": [0], - "expiration": 75, - "unsealed_cid": "bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu", - "seal_proof": "StackedDRG2KiBV1P1" -} -``` - -| Name | Value | Description | -| -------------------------------------------------------------------------------------------------------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------- | -| [`sector_number`](../glossary.md#sector) | 1 | The place where `husky.jpg` will be stored. Charlie decided it'll be on his 1st sector. | -| `deal_ids` | `[0]` | A sector can contain multiple deals, but it only contains the first one ever created (id: 0). | -| `expiration` | `75` | The 75th block is 5 minutes after the `end_block`, so the sector expires only after the deal has been terminated. | -| [`sealed_cid`](../glossary.md#commitment-of-replication), [`unsealed_cid`](../glossary.md#commitment-of-data), | multiple | Currently, placeholder values (any CID) since the proof mechanism is a work-in-progress. | -| `seal_proof` | `StackedDRG2KiBV1P1` | Currently, we only accept sector sizes of 2KiB, so this is the only value possible. | - -`prove-commit-husky.json` - -```json -{ - "sector_number": 1, - "proof": "1230deadbeef" -} -``` - -- `proof`: hex string of bytes of the proof, it's WIP, so any non-zero value works. - -```bash -$ storagext-cli --sr25519-key "//Charlie" storage-provider pre-commit "@pre-commit-husky.json" -[0xf2ad…dc3d] Sector Pre-Committed: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, sector_number: Sector Pre-Commit Info: { sector_number: 1, expiration: 75, seal_proof: RegisteredSealProof::StackedDRG2KiBV1P1, unsealed_cid: bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu, sealed_cid: bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu } } -$ storagext-cli --sr25519-key "//Charlie" storage-provider prove-commit "@prove-commit-husky.json" -[0x0743…6fd3] Sector Proven: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, sector_number: 1, partition_number: 0, deadline_idx: 0 } -``` - - - -[^slash]: Wait for 5 minutes and observe a `DealSlashed` [Event](../pallets/market.md#events) being published. - -## 3. Proofs and faults - -### Aside on Deadlines - -There is a little something that Charlie needs to know about: deadlines _(don't we all...)_. - -Each Storage Provider has a Proving Period, a time divided into segments (deadlines). -To simplify, let's say a proving period lasts a day (24 hours), and between the start and end of each hour, there is a segment, just like on a clock. - - - -Now, when Charlie gathers enough data and stores it in a sector, he must keep proving that he has some data. -Charlie can have multiple sectors and he doesn't want to overload the system by proving all of them simultaneously. -So what if each sector got assigned a segment (1 hour) during the day, and Charlie would need to submit proof that he has data roughly on the same hour each day? -That'd work, right? - -So this is what a Proving Period and Deadlines are. -We divide a proving period into deadlines and when we prove commit, we assign a sector to a deadline. -From now on, the sector must be proven periodically and daily during this lifetime. - -`windowed-post.json` - -```json -{ - "deadline": 0, - "partitions": [0], - "proof": { - "post_proof": "2KiB", - "proof_bytes": "1230deadbeef" - } -} -``` - -| Name | Value | Description | -| ------------- | ---------------- | ----------------------------------------------------------------------------- | -| `deadline` | `0` | the deadline index which has been assigned by the Prove Commit | -| `partitions` | `[0]` | the partitions which have been assigned by Prove Commit | -| `post_proof` | "2KiB" | we only support sectors of size 2KiB for now, so it's the only possible value | -| `proof_bytes` | `0x1230deadbeef` | hex string of bytes of the proof, it's WIP, so any non-zero value works | - -Now that the sector has been proven, Charlie must keep confirming that he stores the data. -Charlie's proving period starts at block `21` (as `register-storage-provider` tells us), so the first deadline is between blocks `[21, 31)`, second `[31, 41)`. -That's because there are [2 deadlines](#const-period-deadlines) within a [proving period](#const-proving-period), and a deadline has a [window of 10 blocks](#const-challenge-window). -Charlie got assigned the first deadline, so he waits until block `21` to send the proof that he still stores the data. - -```bash -$ storagext-cli --sr25519-key "//Charlie" storage-provider submit-windowed-post "@windowed-post.json" -[0x3aa1…a12a] Valid PoSt Submitted: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y } -``` - - - -The next proving period starts at block `41`, with the deadline `[41, 51)`, so Charlie must submit his proof on this block. -He knows he won't be able to create a proof, as his hard drives went down, so he reports it. If he didn't report, he'd get slashed. - -`fault-declaration.json` - -```json -[ - { - "deadline": 0, - "partition": 0, - "sectors": [1] - } -] -``` - -```bash -$ storagext-cli --sr25519-key "//Charlie" storage-provider declare-faults "@fault-declaration.json" -[0x49cd…9a88] Faults Declared: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, faults: [Fault Declaration: { deadline: 0, partition: 0, sectors: [1] }] } -``` - - - -Charlie fixed his issues with storage and now wants to declare that he can still provide data in this sector. -If he does this too late (1 minute before the next deadline starts), he won't be able to. - -```bash -$ storagext-cli --sr25519-key "//Charlie" storage-provider declare-faults-recovered "@fault-declaration.json" -Error: Runtime error: Pallet error: StorageProvider::FaultRecoveryTooLate - -Caused by: - Pallet error: StorageProvider::FaultRecoveryTooLate -``` - -If he does it at least a minute before, it succeeds: - -```bash -$ storagext-cli --sr25519-key "//Charlie" storage-provider declare-faults-recovered "@fault-declaration.json" -[0xca6b…875c] Faults Recovered: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, recoveries: [Recovery Declaration: { deadline: 0, partition: 0, sectors: [1] }] } -``` - -And then, at the next deadline, in the next proving period `[41, 51)` he needs to remember to submit windowed PoSt again. - -```bash -$ storagext-cli --sr25519-key "//Charlie" storage-provider submit-windowed-post "@windowed-post.json" -[0x3e46…5636] Valid PoSt Submitted: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y } -``` - -## 4. Reaping the rewards - -After the deal ends (after block `50`), Charlie goes to get his rewards! -First, he settles all of the locked funds, so his balance gets unlocked, and then he withdraws the balance from his Market account to use his DOTs for a new shiny dumbbell. - -```bash -$ storagext-cli --sr25519-key "//Charlie" market settle-deal-payments 0 -[0x1633…d17b] Deals settled: { successful: [Settled Deal { deal_id: 0, provider_account: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, client_account: 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY, amount: 25000000000 }], unsuccessful: [] } -$ storagext-cli --sr25519-key "//Charlie" market withdraw-balance 37500000000 -[0x0197…bdd9] Balance Withdrawn: { account: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, amount: 37500000000 } -``` diff --git a/src/getting-started/index.md b/src/getting-started/index.md index a552a7b..c722024 100644 --- a/src/getting-started/index.md +++ b/src/getting-started/index.md @@ -11,6 +11,13 @@ Before proceeding with the setup, please ensure the host system meets the follow ## Guides -- [*Local Testnet - Polka Storage Parachain*](local-testnet.md) — Covers how to setup a local testnet for the Polka Storage parachain, using Zombienet. -- [*CAR server*](car-server.md) — Covers how to setup and run the CARv2 server, which transforms files into `.car` archives. -- [*Real-world use case demo*](demo.md) — Covers a real-world use case, adding balances, submitting sectors, and more! +- [_Building Polka Storage_](./building/index.md) + - [_Building from source_](./building/source.md) - Covers how to setup the required dependencies on your machine and build components of the project. + - [_Setting up and building with Docker_](./building/docker.md) - Covers how to setup and build the Docker images for the components of the project. +- [_Local Testnet - Polka Storage Parachain_](local-testnet/index.md) — Covers how to setup a local testnet for the Polka Storage parachain, using Zombienet. + - [_Getting funds_](local-testnet/getting-funds.md) - Covers how to get funds into an account that has been generated externally. +- [_Local Testnet - Polka Storage Parachain_](local-testnet/index.md) — Covers how to setup a local testnet for the Polka Storage parachain, using Zombienet. +- [_Launching a Storage Provider_](storage-provider.md) - Covers how to setup a Storage Provider. +- [_Storing a file_](demo-file-store.md) — Covers how to store a file by the Storage Client. + + diff --git a/src/getting-started/local-testnet/getting-funds.md b/src/getting-started/local-testnet/getting-funds.md new file mode 100644 index 0000000..c33a7b5 --- /dev/null +++ b/src/getting-started/local-testnet/getting-funds.md @@ -0,0 +1,60 @@ +# Getting funds + +This document covers getting funds into an account that has been generated externally. + +## Setting up your account + +In this guide we will be covering getting funds into a externally generated account. The recommended way to generate an account is by using the [polkadot.js wallet extension](https://github.com/polkadot-js/extension). + +Please make sure to follow the instructions on how to generate a new account if you have not done so already. +You can read more about creating a Polkadot account using the extension in the following [link](https://support.polkadot.network/support/solutions/articles/65000098878-how-to-create-a-dot-account#How-to-create-an-account-with-the-Polkadot-extension) + +Or you can watch the following video: + + +## Transferring funds + +Make sure to run the local testnet, you can find how to do so in the [local testnet guide](index.md). Once the local testnet is up and running navigate to the polkadot-js web app interface by going to the [default polkadot.js web interface URL](https://polkadot.js.org/apps/?rpc=ws://127.0.0.1:42069). + +> If you have changed the `ws_port` value in the zombienet configuration — `local-testnet.toml`, +> this URL is different and you should change the port accordingly. + +Under the developer tab, navigate to *Sudo*. + +![sudo selection](../../images/transfer-funds/developer-sudo.png) + +Once you are in *Sudo* you should select `balances` from the submit dropdown. + +![balance selection](../../images/transfer-funds/select-balance.png) + +Then, on the box to the right, select `forceSetBalance` from the dropdown. + +![force selection](../../images/transfer-funds/select-force.png) + +Setting the `Id` field to your generated address and the `newFree` field to the amount in [plancks](../../glossary.md#planck), +as shown in the image below. +If your polkadot.js extension is injected into the polkadot.js web interface it will recognize the injection and you can select the desired account. + +
+ +Note that the `forceSetBalance` extrinsic does **NOT** add tokens to an account but rather **sets the balance** to the given amount. + +
+ +![balance forceSetBalance](../../images/transfer-funds/force-set-balance.png) + +Sign and submit your transaction, the caller will automatically be set to Alice, a dev account. + +![sign and submit](../../images/transfer-funds/sign-and-submit.png) + +After the block has been finalized, the balance will show up in the generated account under the accounts tab and you are ready to start using the polka-storage chain with your own account. + +![account balance](../../images/transfer-funds/account-balance.png) diff --git a/src/getting-started/local-testnet.md b/src/getting-started/local-testnet/index.md similarity index 67% rename from src/getting-started/local-testnet.md rename to src/getting-started/local-testnet/index.md index 42de7a8..d05dadb 100644 --- a/src/getting-started/local-testnet.md +++ b/src/getting-started/local-testnet/index.md @@ -8,13 +8,15 @@ Charlie will be our contact point to the parachain network. ## Native Binaries -Our latest releases's binaries are available to download and can be run without additional dependencies. +The binaries for the latest releases are available to download and can be run without any additional dependencies. We support `Linux x86_64` and `MacOS ARM x64`. The commands below will download: - [Relay Chain](https://github.com/paritytech/polkadot-sdk/releases) binaries (`polkadot`, `polkadot-prepare-worker`, `polkadot-execute-worker`), - Polka Storage Parachain binary (`polka-storage-node`), -- [Polka Storage Provider](../storage-provider-cli/index.md) internal node (`polka-storage-provider`), -- [CLI for interacting with the parachain](../storagext-cli/) (`storagext-cli`), +- [Polka Storage Provider](../../storage-provider-cli/server.md) internal node (`polka-storage-provider-server`), +- [Polka Storage Provider Client](../../storage-provider-cli/client/index.md) internal node's RPC client and proving tools (`polka-storage-provider-client`), +- [Storagext CLI](../../storagext-cli/index.md) to interact with a chain by sending extrinsics (`storagext-cli`), +- [Mater CLI](../../mater-cli/index.md) for CARv2 file archive operations (`mater-cli`), - [zombienet](https://paritytech.github.io/zombienet/install.html) to spawn local testnets and orchestrate them (`zombienet`), - Polka Storage Parachain out-of-the-box zombienet's configuration (`polka-storage-testnet.toml`). @@ -23,19 +25,21 @@ We support `Linux x86_64` and `MacOS ARM x64`. The commands below will download: 1. Download the binaries: ```bash -wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-v1.13.0/polkadot -wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-v1.13.0/polkadot-prepare-worker -wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-v1.13.0/polkadot-execute-worker -wget https://s3.eu-central-1.amazonaws.com/polka-storage/linux_x86-64/polka-storage-node -wget https://s3.eu-central-1.amazonaws.com/polka-storage/linux_x86-64/polka-storage-provider -wget https://s3.eu-central-1.amazonaws.com/polka-storage/linux_x86-64/storagext-cli +wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2407-1/polkadot +wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2407-1/polkadot-prepare-worker +wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2407-1/polkadot-execute-worker +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-node-v0.0.0/polka-storage-node-linux-x86 -O polka-storage-node +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-provider-client-v0.1.0/polka-storage-provider-client-linux-x86 -O polka-storage-provider-client +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-provider-server-v0.1.0/polka-storage-provider-server-linux-x86 -O polka-storage-provider-server +wget https://github.com/eigerco/polka-storage/releases/download/storagext-cli-v0.1.0/storagext-cli-linux-x86 -O storagext-cli +wget https://github.com/eigerco/polka-storage/releases/download/mater-cli-v0.1.0/mater-cli-linux-x86 -O mater-cli wget https://github.com/paritytech/zombienet/releases/download/v1.3.106/zombienet-linux-x64 -O zombienet ``` 2. Setup permissions: ```bash -chmod +x zombienet polka-storage-node polka-storage-provider storagext-cli polkadot polkadot-prepare-worker polkadot-execute-worker +chmod +x zombienet polka-storage-node polka-storage-provider-client polka-storage-provider-server storagext-cli mater-cli polkadot polkadot-prepare-worker polkadot-execute-worker ``` 3. Run `zombienet`: @@ -52,20 +56,22 @@ zombienet -p native spawn polka-storage-testnet.toml 1. Download the binaries: ```bash -wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/polkadot -wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/polkadot-prepare-worker -wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/polkadot-execute-worker -wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/polka-storage-node -wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/polka-storage-provider -wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/storagext-cli +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-node-v0.0.0/polkadot-2407-1-macos-arm64 -O polkadot +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-node-v0.0.0/polkadot-2407-1-prepare-worker-macos-arm64 -O polkadot-prepare-worker +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-node-v0.0.0/polkadot-2407-1-execute-worker-macos-arm64 -O polkadot-execute-worker +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-node-v0.0.0/polka-storage-node-macos-arm64 -O polka-storage-node +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-provider-server-v0.1.0/polka-storage-provider-server-macos-arm64 -O polka-storage-provider-server +wget https://github.com/eigerco/polka-storage/releases/download/polka-storage-provider-client-v0.1.0/polka-storage-provider-client-macos-arm64 -O polka-storage-provider-client +wget https://github.com/eigerco/polka-storage/releases/download/storagext-cli-v0.1.0/storagext-cli-macos-arm64 -O storagext-cli +wget https://github.com/eigerco/polka-storage/releases/download/mater-cli-v0.1.0/mater-cli-macos-arm64 -O mater-cli wget https://github.com/paritytech/zombienet/releases/download/v1.3.106/zombienet-macos-arm64 -O zombienet ``` 2. Setup permissions & de-quarantine: ```bash -chmod +x zombienet polka-storage-node polka-storage-provider storagext-cli polkadot polkadot-prepare-worker polkadot-execute-worker -xattr -d com.apple.quarantine zombienet polka-storage-node polka-storage-provider storagext-cli polkadot polkadot-prepare-worker polkadot-execute-worker +chmod +x zombienet polka-storage-node polka-storage-provider-server polka-storage-provider-client storagext-cli mater-cli polkadot polkadot-prepare-worker polkadot-execute-worker +xattr -d com.apple.quarantine zombienet polka-storage-node polka-storage-provider-server polka-storage-provider-client storagext-cli mater-cli polkadot polkadot-prepare-worker polkadot-execute-worker ```
@@ -90,7 +96,7 @@ The parachain is also accessible using the Polkadot.js Apps interface by clickin

-Or interact with the chain via [`storagext-cli`](../storagext-cli/index.md), for example: +Or interact with the chain via [`storagext-cli`](../../storagext-cli/index.md), for example: ```bash storagext-cli --sr25519-key "//Alice" storage-provider register Alice @@ -131,7 +137,7 @@ image_pull_policy = "IfNotPresent" [relaychain] chain = "rococo-local" default_args = ["--detailed-log-output", "-lparachain=debug,xcm=trace,runtime=trace"] -default_image = "docker.io/parity/polkadot:v1.13.0" +default_image = "docker.io/parity/polkadot:stable2407-1" [[relaychain.nodes]] name = "alice" @@ -153,12 +159,14 @@ id = 1000 [[parachains.collators]] args = ["--detailed-log-output", "-lparachain=debug,xcm=trace,runtime=trace"] command = "polka-storage-node" -image = "polkadotstorage.azurecr.io/parachain-node:0.1.0" +image = "ghcr.io/eigerco/polka-storage-node:0.0.0" name = "charlie" rpc_port = 42069 validator = true ``` +> For details on this configuration, refer to the [Zombienet Configuration](../../zombienet-config.md) chapter. + 2. Run the Parachain, and spawn the zombienet testnet in the Kubernetes cluster: ``` @@ -222,7 +230,9 @@ zombienet -p kubernetes spawn local-kube-testnet.toml Check if all zombienet pods were started successfully: -`kubectl get pods --all-namespaces` +```bash +kubectl get pods --all-namespaces +```
Click here to show the example output. @@ -252,57 +262,66 @@ The parachain is available through the Polkadot.js Apps interface by clicking on This link will automatically connect to Charlie's node running on a local machine at port `42069`. The port is configured in `local-kube-testnet.toml` under `rpc_port` for Charlie's node. -## Zombienet Configuration Breakdown - -During the setup either download a file in the third step of [Linux](#linux-x86_64)/[MacOS](#macos-arm) — `polka-storage-testnet.toml` — -or copy it from the first step of [Running the parachain](#running-the-parachain). - -### Similarities - -The two files share most of the contents, so we'll start by covering their similarities. -For more details refer to the [`zombienet` documentation](https://paritytech.github.io/zombienet/network-definition-spec.html): - -#### `relaychain` +## Checking the logs -| Name | Description | -| ----------------- | --------------------------------------------- | -| `chain` | The relaychain name | -| `default_args` | The default arguments passed to the `command` | -| `default_command` | The default command to run the relaychain | -| `nodes` | List of tables defining the nodes to run | - -##### `nodes` - -| Name | Description | -| ----------- | -------------------------------------- | -| `name` | The node name | -| `validator` | Whether the node is a validator or not | - -#### `parachains` +At the end of the `zombienet` output you should see a table like so: +``` +┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ +│ Node Information │ +├──────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ Name │ charlie │ +├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ Direct Link │ https://polkadot.js.org/apps/?rpc=ws://127.0.0.1:42069#/explorer │ +├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ Prometheus Link │ http://127.0.0.1:44955/metrics │ +├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ Log Cmd │ tail -f │ +│ │ /tmp/nix-shell.gQQj4Y/zombie-bcb786e1748ff0a6becd28289e1f70b9_-677866-G8ea9Qqs65DB/charlie.log │ +├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ Parachain ID │ 1000 │ +├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ ChainSpec Path │ /tmp/nix-shell.gQQj4Y/zombie-bcb786e1748ff0a6becd28289e1f70b9_-677866-G8ea9Qqs65DB/1000-rococo-lo… │ +└──────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘ +``` -A list of tables defining multiple parachains, in our case, we only care for our own parachain. +We strongly recommend you check the logs for the collator (in this case Charlie), using a text editor of your choice. -| Name | Description | -| --------------- | ------------------------------------------------------------- | -| `cumulus_based` | Whether to use `cumulus` based generation | -| `id` | The parachain ID, we're using `1000` as a placeholder for now | -| `collators` | List of tables defining the collators | +> If the **Log Cmd** is shortened with `...`, try to search for the folder using the zombienet namespace, available at the top of the table. +> For example: +> ```bash +> $ ls /tmp | grep zombie-bcb786e1748ff0a6becd28289e1f70b9 +> ``` -##### `collators` +After finding the `charlie.log` file, you should `grep` for errors relating to the extrinsic you just ran, +for example, if you ran a `market` extrinsic, like `add-balance`, you would run: -| Name | Description | -| ----------- | ---------------------------------------- | -| `args` | The arguments passed to the `command` | -| `command` | The command to run the collator | -| `name` | The collator name | -| `validator` | Whether the collator is also a validator | +```bash +$ grep "ERROR.*runtime::market" charlie.log +``` -### Differences +Or, more generally: -The difference between them lies in the usage of container configurations: +```bash +$ grep ".*runtime::" charlie.log +``` -| Name | Description | -| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `image_pull_policy` | Defines when `zombienet` should pull an image; read more about it in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy) | -| `image` | Defines which image to pull | -| `ws_port`/`rpc_port` | Depending on the type of configuration (Native or Kubernetes), this variable sets the port for the collator RPC service | +Where `LOG_LEVEL` is one of: +* `DEBUG` +* `INFO` +* `WARN` +* `ERROR` + +And the extrinsic pallet, is one of: +* `storage_provider` +* `market` +* `proofs` + +> **Tip:** if you are running into the `AllProposalsInvalid` error, +> try searching for `insane deal` in the logs, you should find the cause faster! +> +> For example: +> ```bash +> $ grep "insane deal" -B 1 charlie.log +> 2024-11-14 13:24:24.019 ERROR tokio-runtime-worker runtime::market: [Parachain] deal duration too short: 100 < 288000 +> 2024-11-14 13:24:24.019 ERROR tokio-runtime-worker runtime::market: [Parachain] insane deal: idx 0, error: ProposalError::DealDurationOutOfBounds +> ``` diff --git a/src/getting-started/storage-provider.md b/src/getting-started/storage-provider.md new file mode 100644 index 0000000..b6f3d8a --- /dev/null +++ b/src/getting-started/storage-provider.md @@ -0,0 +1,107 @@ +# Launching the Storage Provider + +> This guide assumes you have read the [Building](./building/index.md) +> and the [Local Testnet - Polka Storage](./local-testnet/index.md) guides +> and have a running testnet to connect to. + +Setting up the Storage Provider doesn't have a lot of science, but isn't automatic either! +In this guide, we'll cover how to get up and running with the Storage Provider. + +## Generating the PoRep Parameters + +First and foremost, to allow the Storage Provider to generate [PoRep](https://docs.filecoin.io/basics/the-blockchain/proofs#proof-of-replication-porep) proofs, +we need to first generate their parameters, we do that with the following command: + +```bash +$ polka-storage-provider-client proofs porep-params +Generating params for 2KiB sectors... It can take a couple of minutes ⌛ +Generated parameters: +/home/user/polka-storage/2KiB.porep.params +/home/user/polka-storage/2KiB.porep.vk +/home/user/polka-storage/2KiB.porep.vk.scale +``` + +As advertised, the command has generated the following files: + +* `2KiB.porep.params` — The PoRep parameters +* `2KiB.porep.vk` — The verifying key +* `2KiB.porep.vk.scale` — The verifying key, encoded in SCALE format + +## Registering the Storage Provider + +> If you encounter errors while running extrinsics, check the parachain logs. +> Refer to the [Checking the logs](./local-testnet/index.md#checking-the-logs) section under the +> [Local Testnet - Polka Storage Provider](./local-testnet/index.md) chapter. + +Logically, if you want to participate in the network, you need to register. +To do so, you need to run one of the following commands: + +```bash +storagext-cli --sr25519-key storage-provider register "" +storagext-cli --ed25519-key storage-provider register "" +storagext-cli --ecdsa-key storage-provider register "" +``` + + +Where `` has been replaced accordingly to its key type. +`` can be anything as it is currently used as a placeholder. *For example:* +``` +storagext-cli --sr25519-key "//Charlie" storage-provider register "placeholder" +``` + +After registering, there is one more thing to be done, to be able to verify proofs in local testnet. +We need to set the global verifying key in the network, so it's compatible with the proving parameters: + +```bash +storagext-cli --sr25519-key "//Charlie" proofs set-porep-verifying-key @2KiB.porep.vk.scale +``` + +> Additionally, you will need to add some balance to your Polka Storage escrow account, like so: +> ``` +> $ storagext-cli --sr25519-key "//Charlie" market add-balance 12500000000 +> [0x809d…8f10] Balance Added: { account: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, amount: 12500000000 } +> ``` +> You can use other balance values! There's a minimum though — `1_000_000_000` (without the `_`). + +And you're ready! + +## Launching the server 🚀 + +Similarly to the previous steps, here too you'll need to run a command. +The following is the *minimal* command: + +```bash +polka-storage-provider-server \ + --seal-proof 2KiB \ + --post-proof 2KiB \ + --porep-parameters \ + --X-key +``` + +Where `--X-key ` matches the key type you used to register yourself with the network, in the previous step. For example: +```bash +polka-storage-provider-server \ + --seal-proof 2KiB \ + --post-proof 2KiB \ + --porep-parameters "2KiB.porep.params" \ + --sr25519-key "//Charlie" +``` + +Note that currently, `--seal-proof` and `--post-proof` only support `2KiB`. + +`` is the resulting `*.porep.params` file from the [first steps](#generating-the-porep-parameters), +in this case, `2KiB.porep.params`. + +When ran like this, the server will assume a random directory for the database and the storage, however, +you can change that through the `--database-directory` and `--storage-directory`, respectively, +if the directory does not exist, it will be created. + +You can also change the parachain node address it connects to, +by default, the server will try to connect to `ws://127.0.0.1:42069`, +but you can change this using the `--node-url` flag. + +Finally, you can change the listening addresses for the RPC and HTTP services, +they default to `127.0.0.1:8000` and `127.0.0.1:8001` respectively, +and can be changed using the flags `--rpc-listen-address` and `--upload-listen-address`. + +For more information on the available flags, refer to the [`server` chapter](../storage-provider-cli/server.md). diff --git a/src/images/architecture/collator_overview.svg b/src/images/architecture/collator_overview.svg new file mode 100644 index 0000000..c1644f4 --- /dev/null +++ b/src/images/architecture/collator_overview.svg @@ -0,0 +1,172 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/images/architecture/pallets_overview.png b/src/images/architecture/pallets_overview.png new file mode 100644 index 0000000..b12f400 Binary files /dev/null and b/src/images/architecture/pallets_overview.png differ diff --git a/src/images/architecture/storage-provider-pipeline.png b/src/images/architecture/storage-provider-pipeline.png new file mode 100644 index 0000000..3244560 Binary files /dev/null and b/src/images/architecture/storage-provider-pipeline.png differ diff --git a/src/images/architecture/storage-provider-server.png b/src/images/architecture/storage-provider-server.png new file mode 100644 index 0000000..b3773d8 Binary files /dev/null and b/src/images/architecture/storage-provider-server.png differ diff --git a/src/images/architecture/storage-provider-server.svg b/src/images/architecture/storage-provider-server.svg new file mode 100644 index 0000000..70316de --- /dev/null +++ b/src/images/architecture/storage-provider-server.svg @@ -0,0 +1,109 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/images/architecture/storage_provider_overview.svg b/src/images/architecture/storage_provider_overview.svg new file mode 100644 index 0000000..1fc14b1 --- /dev/null +++ b/src/images/architecture/storage_provider_overview.svg @@ -0,0 +1,122 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/images/architecture/system_overview.png b/src/images/architecture/system_overview.png new file mode 100644 index 0000000..9e68843 Binary files /dev/null and b/src/images/architecture/system_overview.png differ diff --git a/src/images/architecture/system_overview.svg b/src/images/architecture/system_overview.svg new file mode 100644 index 0000000..a19f5c4 --- /dev/null +++ b/src/images/architecture/system_overview.svg @@ -0,0 +1,116 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/images/background/challenge_protocol.png b/src/images/background/challenge_protocol.png new file mode 100644 index 0000000..86d4384 Binary files /dev/null and b/src/images/background/challenge_protocol.png differ diff --git a/src/images/background/merkle_tree_comparison.png b/src/images/background/merkle_tree_comparison.png new file mode 100644 index 0000000..2bba1e6 Binary files /dev/null and b/src/images/background/merkle_tree_comparison.png differ diff --git a/src/images/background/merkle_tree_storage_provider.png b/src/images/background/merkle_tree_storage_provider.png new file mode 100644 index 0000000..89c63cc Binary files /dev/null and b/src/images/background/merkle_tree_storage_provider.png differ diff --git a/src/images/background/merkle_tree_verifier.png b/src/images/background/merkle_tree_verifier.png new file mode 100644 index 0000000..cac72e9 Binary files /dev/null and b/src/images/background/merkle_tree_verifier.png differ diff --git a/src/images/polkadot.svg b/src/images/polkadot.svg new file mode 100644 index 0000000..9d5bad6 --- /dev/null +++ b/src/images/polkadot.svg @@ -0,0 +1,14 @@ + + + diff --git a/src/images/showcase/readme.md b/src/images/showcase/README.md similarity index 59% rename from src/images/showcase/readme.md rename to src/images/showcase/README.md index 48e3ac2..0198916 100644 --- a/src/images/showcase/readme.md +++ b/src/images/showcase/README.md @@ -3,12 +3,12 @@ The tools used for the showcase are [asciinema](https://asciinema.org/), [asciin Steps to record a new asciinema session: 1. Install asciinema or use the docker image -2. Start the Local Testnet - Polka Storage Parachain -3. Update the `cli_basic.sh` with the commands you want to showcase +2. Check the README.md of the specific showcase for any additional steps +3. Update the `showcase.sh` with the commands you want to showcase 4. Start the asciinema recording - `asciinema-automation --timeout 300 ./cli_basic.sh prod.cast` + `asciinema-automation --timeout 300 ./showcase.sh showcase.cast` 5. Convert the `.cast` to `.gif` - `agg prod.cast prod.gif` + `agg showcase.cast showcase.gif` diff --git a/src/images/showcase/cli_basic.gif b/src/images/showcase/cli_basic.gif deleted file mode 100644 index 09295f8..0000000 Binary files a/src/images/showcase/cli_basic.gif and /dev/null differ diff --git a/src/images/showcase/cli_basic.sh b/src/images/showcase/cli_basic.sh deleted file mode 100644 index a804a86..0000000 --- a/src/images/showcase/cli_basic.sh +++ /dev/null @@ -1,60 +0,0 @@ -# This script demonstrates the basic workflow of a storage provider. It is used -# by the asciinema_automation to automate the video session. It includes -# registering a provider, adding balances, publishing storage deals, -# pre-committing sectors, and proving commitments. It uses the storagext-cli -# tool to interact with the network. - -# The script expecs the storagext-cli tool to be in the current directory. - -# mean of gaussian delay between key strokes, default to 50ms -#$ delay 20 - -./storagext-cli --sr25519-key "//Alice" \ - market add-balance 1000000000000 -#$ expect Balance Added - -./storagext-cli --sr25519-key "//Charlie" \ - market add-balance 1000000000000 -#$ expect Balance Added - -./storagext-cli --sr25519-key "//Alice" \ - storage-provider register alice -#$ expect Storage Provider Registered - -./storagext-cli --sr25519-key "//Alice" \ - market publish-storage-deals --client-sr25519-key "//Charlie" ' -[ - { - "piece_cid": "bafk2bzacecg3xxc4f2ql2hreiuy767u6r72ekdz54k7luieknboaakhft5rgk", - "piece_size": 1337, - "client": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", - "provider": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", - "label": "Awesome piece", - "start_block": 100, - "end_block": 200, - "storage_price_per_block": 15, - "provider_collateral": 2000, - "state": "Published" - } -]' -#$ expect Deal Published - -./storagext-cli --sr25519-key "//Alice" \ - storage-provider pre-commit ' -{ - "sector_number": 0, - "sealed_cid": "bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu", - "deal_ids": [0], - "expiration": 200, - "unsealed_cid": "bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu", - "seal_proof": "StackedDRG2KiBV1P1" -}' -#$ expect Sector Pre-Committed - -./storagext-cli --sr25519-key "//Alice" \ - storage-provider prove-commit ' -{ - "sector_number": 0, - "proof": "64756D6D792070726F6F66" -}' -#$ expect Sector Proven diff --git a/src/images/showcase/client_upload/README.md b/src/images/showcase/client_upload/README.md new file mode 100644 index 0000000..c5ebf38 --- /dev/null +++ b/src/images/showcase/client_upload/README.md @@ -0,0 +1,12 @@ +### Prerequisites + +Script expects `mater-cli`, `polka-storage-provider-client`, `polka-storage-provider-server`, `storagext-cli` in the current directory. + +1. Start zombienet and call required extrinsics + - Setup Alices balances `./storagext-cli --sr25519-key //Alice market add-balance 250000000000` + - Setup Charlies balances `./storagext-cli --sr25519-key //Charlie market add-balance 250000000000` + - Register Charlie as a storage provider `./storagext-cli --sr25519-key "//Charlie" storage-provider register "peer_id"` +2. Generate proving params + - `./polka-storage-provider-client proofs porep-params` +3. Start storage provider server + - `./polka-storage-provider-server --sr25519-key //Charlie --seal-proof "2KiB" --post-proof "2KiB" --porep-parameters 2KiB.porep.params` diff --git a/src/images/showcase/client_upload/data.txt b/src/images/showcase/client_upload/data.txt new file mode 100644 index 0000000..5855f49 --- /dev/null +++ b/src/images/showcase/client_upload/data.txt @@ -0,0 +1,158 @@ +1234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +12341234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +1234 +1234567891 +1234567891 +1234567891 +1234567891 +1234567891 +1234567891 +1234567891 +1234567891 +1234567891 +123 \ No newline at end of file diff --git a/src/images/showcase/client_upload/deal.json b/src/images/showcase/client_upload/deal.json new file mode 100644 index 0000000..5fb8cc6 --- /dev/null +++ b/src/images/showcase/client_upload/deal.json @@ -0,0 +1,12 @@ +{ + "piece_cid": "baga6ea4seaqbfhdvmk5qygevit25ztjwl7voyikb5k2fqcl2lsuefhaqtukuiii", + "piece_size": 2048, + "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", + "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", + "label": "custom label", + "start_block": 200, + "end_block": 250, + "storage_price_per_block": 500, + "provider_collateral": 1250, + "state": "Published" +} \ No newline at end of file diff --git a/src/images/showcase/client_upload/showcase.gif b/src/images/showcase/client_upload/showcase.gif new file mode 100644 index 0000000..909b100 Binary files /dev/null and b/src/images/showcase/client_upload/showcase.gif differ diff --git a/src/images/showcase/client_upload/showcase.sh b/src/images/showcase/client_upload/showcase.sh new file mode 100644 index 0000000..ce14faf --- /dev/null +++ b/src/images/showcase/client_upload/showcase.sh @@ -0,0 +1,45 @@ +# This script demonstrates the process of uploading data to Polka Storage, +# creating and signing a storage deal, and publishing it on the network. It is +# used by the asciinema_automation to automate the video session. +# +# The script performs the following steps: +# 1. Converts input data to CAR format +# 2. Generates CommP (Piece CID) for the data +# 3. Creates a storage deal JSON +# 4. Signs the storage deal +# 5. Proposes the deal and obtains a Deal CID +# 6. Uploads the data to the storage provider +# 7. Publishes the signed deal on the network + +# mean of gaussian delay between key strokes, default to 50ms +#$ delay 20 + +./mater-cli convert -q data.txt data.car +#$ expect + +./polka-storage-provider-client proofs commp data.car +#$ expect cid + +DEAL_JSON="$(cat ./deal.json)" +echo "$DEAL_JSON" + +SIGNED_DEAL_JSON="$(./polka-storage-provider-client sign-deal --sr25519-key //Alice "$DEAL_JSON")" +#$ expect + +echo "$SIGNED_DEAL_JSON" +#$ expect client_signature + +DEAL_CID="$(./polka-storage-provider-client propose-deal "$DEAL_JSON")" +#$ expect + +echo "$DEAL_CID" +#$ expect bagaaieravcqzkt2ilbdghlw3metiwuuqklfq2udxiubsghyj47ha3wuogppq + +curl -X PUT -F "upload=@data.txt" "http://localhost:8001/upload/$DEAL_CID" +#$ expect baga6ea4seaqbfhdvmk5qygevit25ztjwl7voyikb5k2fqcl2lsuefhaqtukuiii + +# wait instructions change the time between instructions, default to 80ms +#$ wait 1000 + +./polka-storage-provider-client publish-deal "$SIGNED_DEAL_JSON" +#$ expect 0 diff --git a/src/images/storage-provider/sector_activation.mermaid b/src/images/storage-provider/sector_activation.mermaid new file mode 100644 index 0000000..2b267a4 --- /dev/null +++ b/src/images/storage-provider/sector_activation.mermaid @@ -0,0 +1,8 @@ +sequenceDiagram + Storage Provider->>+Randomness Pallet: get_randomness(seal_block) + Storage Provider->>+Storage Provider Pallet: pre_commit + Storage Provider Pallet->>+Market Pallet: verify_deals + Storage Provider->>+Randomness Pallet: get_porep_randomness(precommit_block) + Storage Provider->>+Storage Provider Pallet: prove_commit_sectors + Storage Provider Pallet->>+Proofs Pallet: verify_porep + Storage Provider Pallet->>+Market Pallet: activate_deals \ No newline at end of file diff --git a/src/images/storage-provider/sector_activation.svg b/src/images/storage-provider/sector_activation.svg index 4434efb..a51f543 100644 --- a/src/images/storage-provider/sector_activation.svg +++ b/src/images/storage-provider/sector_activation.svg @@ -1,3 +1,3 @@ -Market PalletStorage Provider PalletStorage Providerpre_commit_sectorverify_dealsprove_commit_sectoractivate_deals +Proofs PalletMarket PalletStorage Provider PalletRandomness PalletStorage ProviderProofs PalletMarket PalletStorage Provider PalletRandomness PalletStorage Providerget_randomness(seal_block)pre_commitverify_dealsget_porep_randomness(precommit_block)prove_commit_sectorsverify_porepactivate_deals \ No newline at end of file diff --git a/src/images/transfer-funds/account-balance.png b/src/images/transfer-funds/account-balance.png new file mode 100644 index 0000000..d6f9e52 Binary files /dev/null and b/src/images/transfer-funds/account-balance.png differ diff --git a/src/images/transfer-funds/developer-sudo.png b/src/images/transfer-funds/developer-sudo.png new file mode 100644 index 0000000..9be3519 Binary files /dev/null and b/src/images/transfer-funds/developer-sudo.png differ diff --git a/src/images/transfer-funds/force-set-balance.png b/src/images/transfer-funds/force-set-balance.png new file mode 100644 index 0000000..ec8f397 Binary files /dev/null and b/src/images/transfer-funds/force-set-balance.png differ diff --git a/src/images/transfer-funds/select-balance.png b/src/images/transfer-funds/select-balance.png new file mode 100644 index 0000000..b403f6a Binary files /dev/null and b/src/images/transfer-funds/select-balance.png differ diff --git a/src/images/transfer-funds/select-force.png b/src/images/transfer-funds/select-force.png new file mode 100644 index 0000000..fb16f66 Binary files /dev/null and b/src/images/transfer-funds/select-force.png differ diff --git a/src/images/transfer-funds/sign-and-submit.png b/src/images/transfer-funds/sign-and-submit.png new file mode 100644 index 0000000..0c371ba Binary files /dev/null and b/src/images/transfer-funds/sign-and-submit.png differ diff --git a/src/introduction.md b/src/introduction.md index 7020a97..b44c0b0 100644 --- a/src/introduction.md +++ b/src/introduction.md @@ -4,38 +4,70 @@ Welcome to the Polka Storage project! This project aims to deliver a Polkadot-native system parachain for data storage. -Since the Referendum approval, we've been busy developing the parachain, -and this is our deliverable for Phase 1, composed of: - -- Filecoin actor ports: - - [Storage Provider](./pallets/storage-provider.md) — excluding proving mechanisms - - [Market](./pallets/market.md) -- [CAR file conversion server](./storage-provider-cli/storage.md) -- Dedicated CLIs - - [`storage-provider-cli`](./storage-provider-cli/storage.md) to generate keys and test our CARv2 Rust implementation! - - [`storagext-cli`](./storagext-cli/index.md) (shown below) to take the parachain for a spin! +Since the Referendum approval, we've been busy developing the deliverables for Phase 2. + +**For [**Phase 2**](https://polkadot.polkassembly.io/referenda/1150), we have implemented:** + +- Storage Provider Pallet + - [`terminate_sectors`](./architecture/pallets/storage-provider.md#terminate_sectors) + - `process_early_terminations` (internal) + - Batch support + - [Pre-commit](./architecture/pallets/storage-provider.md#pre_commit_sectors) + - [Prove-commit](./architecture/pallets/storage-provider.md#prove_commit_sectors) +- [Storage Provider Server](./architecture/polka-storage-provider-server.md) + - [File upload](./getting-started/demo-file-store.md) + - [Proving Pipeline](./architecture/polka-storage-provider-server.md#sealing-pipeline) + - Pre-commit + - PoRep +- On-chain proof validation + - Proof of Replication + - Proof of Spacetime + + + +We also present a complete [real-world scenario](./getting-started/demo-file-store.md) in which a [Storage Provider](./glossary.md#storage-provider) and a [Storage User](./glossary.md#storage-user) negotiate a deal, perform all the steps necessary to start the storage and then receive rewards (or punishments) for making it happen. + +**The Polka Storage project currently provides:** + +Dedicated CLIs + +- [Polka Storage Provider CLI](./storage-provider-cli/index.md) + - [`polka-storage-provider-server`](./storage-provider-cli/server.md) to launch the Storage Provider server. + - [`polka-storage-provider-client`](./storage-provider-cli/client/index.md) to manage the wallet, propose & publish deals and do proof demos. +- [`mater-cli`](./mater-cli/index.md) to convert or extract CARv2 files. +- [`storagext-cli`](./storagext-cli/index.md) to interact **directly** with the parachain — watch out, this is a low-level tool! + +Filecoin actor ports: + +- [Storage Provider](./architecture/pallets/storage-provider.md) +- [Market](./architecture/pallets/market.md) +- [Proofs](./architecture/pallets/proofs.md) +- [Randomness](./architecture/pallets/randomness.md) +

Polka Storage CLI tooling showcase + src="images/showcase/client_upload/showcase.gif" + alt="Polka Storage Client Upload">

-The following on-chain logic has been implemented: +**As a refresher, for [Phase 1](https://polkadot.polkassembly.io/referenda/494), we implemented the following:** - Keeping track of [Storage Providers](./glossary.md#storage-provider), -- [Publishing](./pallets/market.md#publish_storage_deals) Market Deals on-chain, -- [Investing](./pallets/market.md#add_balance) tokens into the Storage Market, -- [Receiving](./pallets/market.md#settle_deal_payments) funds after completing a deal, -- [Commiting](./pallets/storage-provider.md#pre_commit_sector) to the Storage and [Proving](./pallets/storage-provider.md#prove_commit_sector) the storage, -- [Declaring](./pallets/storage-provider.md#prove_commit_sector) failures to deliver committed storage and [Recovering](./pallets/storage-provider.md#declaring-storage-faults-recovered) from it, -- [Continuously proving](./pallets/storage-provider.md#submit_windowed_post) that the promise of storage has been kept up [PoSt proof](./glossary.md#proofs), -- [Punishing](./pallets/storage-provider.md#events) for failing to provide storage. +- [Publishing](./architecture/pallets/market.md#publish_storage_deals) Market Deals on-chain, +- [Investing](./architecture/pallets/market.md#add_balance) tokens into the Storage Market, +- [Receiving](./architecture/pallets/market.md#settle_deal_payments) funds after completing a deal, +- [Committing](./architecture/pallets/storage-provider.md#pre_commit_sectors) to the Storage and [Proving](./architecture/pallets/storage-provider.md#prove_commit_sectors) the storage, +- [Declaring](./architecture/pallets/storage-provider.md#prove_commit_sectors) failures to deliver committed storage and [Recovering](./architecture/pallets/storage-provider.md#declaring-storage-faults-recovered) from it, +- [Continuously proving](./architecture/pallets/storage-provider.md#submit_windowed_post) that the promise of storage has been kept up [PoSt proof](./glossary.md#proofs), +- [Terminating](./architecture/pallets/storage-provider.md#terminate_sectors) sectors by the storage provider, +- [Punishing](./architecture/pallets/storage-provider.md#events) for failing to provide storage. -We present a complete [real-world scenario](./getting-started/demo.md) in which a [Storage Provider](./glossary.md#storage-provider) and a [Storage User](./glossary.md#storage-user) negotiate a deal, perform all the steps necessary to start the storage and then receive rewards (or punishments) for making it happen. +We present a demo on how to [store a file](./getting-started/demo-file-store.md), where a [Storage Provider](./glossary.md#storage-provider) and a [Storage User](./glossary.md#storage-user) negotiate a deal and perform all the steps necessary to start the file storage. We cover the details behind proving a file in a [separate demo](./getting-started/demo-file-prove.md). -More information available about the project's genesis in: +**More information available about the project's genesis in:** -- OpenGov Referendum — +- OpenGov Referendum - Part 1 — +- OpenGov Referendum - Part 2 — - Research Report — - Polkadot Forum News Post — diff --git a/src/mater-cli/index.md b/src/mater-cli/index.md new file mode 100644 index 0000000..0f5fef1 --- /dev/null +++ b/src/mater-cli/index.md @@ -0,0 +1,84 @@ +# Mater CLI + + + +The Mater CLI is used by storage clients to convert files to the CARv2 format and extract CARv2 content. + +> Currently, the `mater-cli` only supports the CARv2 format. +> However, _the `mater` library has full support for CARv1_. + +To learn more about the CAR format, please refer to the official specifications: + +- CARv1 — +- CARv2 — + +## `convert` + +The convert command converts a file to CARv2 format. + +`mater-cli convert [OUTPUT_PATH]` + +| Argument | Description | +| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| `` | Path to input file | +| `[OUTPUT_PATH]` | Optional path to output CARv2 file. If no output path is given it will store the `.car` file in the same location as the input file. | +| `-q`/`--quiet` | If enabled, only the resulting CID will be printed. | +| `--overwrite` | If enabled, the output will overwrite any existing files. | + +### Example + +```bash +$ mater-cli convert random1024.piece +Converted examples/random1024.piece and saved the CARv2 file at examples/random1024.car with a CID of bafkreidvyofebclo4kny43vpoe5kejg3mqtpq2eemaojzyvlwikwdvusxy +``` + +You can verify the output file using [`go-car`](https://github.com/ipld/go-car): + +```bash +$ car inspect examples/random1024.car +Version: 2 +Characteristics: 00000000000000000000000000000000 +Data offset: 51 +Data (payload) length: 1121 +Index offset: 1172 +Index type: car-multihash-index-sorted +Roots: bafkreidvyofebclo4kny43vpoe5kejg3mqtpq2eemaojzyvlwikwdvusxy +Root blocks present in data: Yes +Block count: 1 +Min / average / max block length (bytes): 1024 / 1024 / 1024 +Min / average / max CID length (bytes): 36 / 36 / 36 +Block count per codec: + raw: 1 +CID count per multihash: + sha2-256: 1 +``` + +## `extract` + +Convert a CARv2 file to its original format. + +`mater-cli extract [OUTPUT_PATH]` + +| Argument | Description | +| --------------- | ------------------------------------------------------------------------------------------------------------------------------ | +| `` | Path to CARv2 file | +| `[OUTPUT_PATH]` | Optional path to output file. If no output path is given it will remove the extension and store the file in the same location. | + +### Example + +```bash +$ mater-cli extract examples/random1024.car +Successfully converted CARv2 file examples/random1024.car and saved it to to examples/random1024 +``` + +Conversly, you can also extract files generated using `car`: + +```bash +# --no-wrap is necessary since mater does not perform wrapping +$ car create --no-wrap -f examples/random1024.go.car examples/random1024.piece +``` + +```bash +$ cargo run -r --bin mater-cli extract examples/random1024.go.car +Successfully converted CARv2 file examples/random1024.go.car and saved it to to examples/random1024.go +``` diff --git a/src/pallets/index.md b/src/pallets/index.md deleted file mode 100644 index e4e0742..0000000 --- a/src/pallets/index.md +++ /dev/null @@ -1,52 +0,0 @@ -# Polka Storage pallets - -- [`storage-provider`](storage-provider.md) - A pallet that manages storage providers and their associated data. -- [`market`](market.md) - A pallet that handles the storage market operations. - -## Overview - -The Polka Storage parachain is all about making storage deals. Let us go over how a deal is done! - -Before anything else, [Storage Providers](../glossary.md#storage-provider) need to register themselves with the [Storage Provider Pallet](./storage-provider.md) — they can do so using the [`register_storage_provider`](./storage-provider.md#register_storage_provider) extrinsic. - -Storage Provider registration - -Now that storage providers can be registered in the storage provider pallet, we need to add some balance to both the -[Storage User's](../glossary.md#storage-user) and the Provider's accounts, -which is done using the Market's [`add_balance`](./market.md#add_balance) extrinsic. - -Adding balance to Market accounts - -Afterwards, storage users and providers negotiate data storage deals off-chain. -Once a deal between the two parties is reached, the client can sign the deal and send it to the storage provider for publishing -— the storage provider will then publish the signed deal using the [`publish_storage_deals`](market.md#publish_storage_deals) extrinsic. - -After publishing, the funds allocated for the deal will be moved from `free` to `locked`, and they can no longer be withdrawn until the deal has ended. - -Publishing storage deals - -At this point, the remaining responsibility is shifted to the storage provider, which needs to activate the deal; -to do so, the storage provider first needs to [pre-commit](./storage-provider.md#pre_commit_sector) the deal's sectors, -and then [prove](./storage-provider.md#prove_commit_sector) they stored the sectors -— these two steps are done using the [`pre_commit_sector`](./storage-provider.md#pre_commit_sector) and [`prove_commit_sector`](./storage-provider.md#prove_commit_sector) extrinsics. - -Verification is done and reported to the Market pallet to terminate the deal and apply penalties to the storage provider -(remove and burn its collateral — i.e. `locked` funds) if they fail to activate the deal on time and return the funds to the client. - -Deal activation - -Suppose the deal has been completed successfully or is **Active**. -In that case, the storage provider is now required to periodically submit proofs that they're still storing the user's data -— the storage provider does this by calculating a proof and submitting it using [`submit_windowed_post`](./storage-provider.md#submit_windowed_post). - -Proving the data is still stored - -Finally, storage providers can then settle deal payments to receive their fair share for keeping the user's data safe — using the [`settle_deal_payments`](./market.md#settle_deal_payments) extrinsic. - -Settling deal payments - -Putting it all together, we get the following: - -The described flow - - diff --git a/src/storage-provider-cli/client/index.md b/src/storage-provider-cli/client/index.md new file mode 100644 index 0000000..518b990 --- /dev/null +++ b/src/storage-provider-cli/client/index.md @@ -0,0 +1,116 @@ +# client + +We cover the commands provided by the `polka-storage-provider-client` CLI tool. + +## `wallet` + +The `wallet` command is a thin wrapper over the [`subkey`](https://docs.substrate.io/reference/command-line-tools/subkey/) utility provided by Polkadot. + +More information available on the [`wallet`](./wallet.md) page. + +## `info` + +The `info` command retrieves information about the storage provider it connects to. + +```bash +$ polka-storage-provider-client info --rpc-server-url "http://127.0.0.1:8000" +{ + "start_time": "2024-11-06T11:29:06.058967136Z", + "address": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", + "seal_proof": "StackedDRG2KiBV1P1", + "post_proof": "StackedDRGWindow2KiBV1P1" +} +``` + +## `propose-deal` + +The `propose-deal` command sends an *unsigned* deal to the storage provider, +if the storage provider accepts the deal, a CID will be returned, +that CID can then be used to upload a file to the storage provider — +for details on this process, refer to the [File Upload chapter](../../getting-started/demo-file-store.md). + +> For the current MVP, the storage provider accepts all valid deals! + +```bash +$ DEAL_TO_PROPOSE='{ + "piece_cid": "baga6ea4seaqj527iqfb2kqhy3tmpydzroiigyaie6g3txai2kc3ooyl7kgpeipi", + "piece_size": 2048, + "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", + "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", + "label": "", + "start_block": 200, + "end_block": 250, + "storage_price_per_block": 500, + "provider_collateral": 1250, + "state": "Published" +}' +# when we omit the `--rpc-server-address` it defaults to "http://127.0.0.1:8000" +$ polka-storage-provider-client propose-deal "$DEAL_TO_PROPOSE" +bagaaieradsfmawozrmgjwxosarexpg7w7ytoe7xw2c63hv6svdc5hpucqo3a +``` + +## `sign-deal` + +The `sign-deal` commands takes a deal like the one passed to [`propose-deal`](#propose-deal) and signs it using the passed key, +the returned deal can then be used with [`publish-deal`](#publish-deal) to send a deal for publishing. +*This command does not call out to the network.* + +```bash +$ DEAL_TO_SIGN='{ + "piece_cid": "baga6ea4seaqj527iqfb2kqhy3tmpydzroiigyaie6g3txai2kc3ooyl7kgpeipi", + "piece_size": 2048, + "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", + "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", + "label": "", + "start_block": 200, + "end_block": 250, + "storage_price_per_block": 500, + "provider_collateral": 1250, + "state": "Published" +}' +$ polka-storage-provider-client sign-deal --sr25519-key "//Charlie" "$DEAL_TO_SIGN" +{ + "deal_proposal": { + "piece_cid": "baga6ea4seaqj527iqfb2kqhy3tmpydzroiigyaie6g3txai2kc3ooyl7kgpeipi", + "piece_size": 2048, + "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", + "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", + "label": "", + "start_block": 200, + "end_block": 250, + "storage_price_per_block": 500, + "provider_collateral": 1250, + "state": "Published" + }, + "client_signature": { + "Sr25519": "32809cd5b53fa3c2e977f77c4e2189dee230b8773946cf94a704f8af19c578289c11ad256b56146195cfc5d7bb8f670003e4575e133f799f19696495046ed58f" + } +} +``` + +## `publish-deal` + +The `publish-deal` command effectively publishes the deal, its input is a deal signed using [`sign-deal`](#sign-deal), +and the output is the on-chain deal ID. + +```bash +$ SIGNED_DEAL='{ + "deal_proposal": { + "piece_cid": "baga6ea4seaqj527iqfb2kqhy3tmpydzroiigyaie6g3txai2kc3ooyl7kgpeipi", + "piece_size": 2048, + "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", + "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y", + "label": "", + "start_block": 200, + "end_block": 250, + "storage_price_per_block": 500, + "provider_collateral": 1250, + "state": "Published" + }, + "client_signature": { + "Sr25519": "32809cd5b53fa3c2e977f77c4e2189dee230b8773946cf94a704f8af19c578289c11ad256b56146195cfc5d7bb8f670003e4575e133f799f19696495046ed58f" + } +}' +$ polka-storage-provider-client publish-deal "$SIGNED_DEAL" +0 +``` diff --git a/src/storage-provider-cli/client/proofs.md b/src/storage-provider-cli/client/proofs.md new file mode 100644 index 0000000..4edb913 --- /dev/null +++ b/src/storage-provider-cli/client/proofs.md @@ -0,0 +1,15 @@ +# `proofs` + +The following subcommands are contained under `proofs`. + +> These are advanced commands and only useful for demo purposes. +> This functionality is covered in the server by the [pipeline](../../architecture/polka-storage-provider-server.md#sealing-pipeline). + +| Name | Description | +| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | +| `calculate-piece-commitment` | Calculate a piece commitment for the provided data stored at the a given path | +| `porep-params` | Generates PoRep verifying key and proving parameters for zk-SNARK workflows (prove commit) | +| `post-params` | Generates PoSt verifying key and proving parameters for zk-SNARK workflows (submit windowed PoSt) | +| `porep` | Generates PoRep for a piece file. Takes a piece file (in a CARv2 archive, unpadded), puts it into a sector (temp file), seals and proves it | +| `post` | Creates a PoSt for a single sector | + diff --git a/src/storage-provider-cli/wallet.md b/src/storage-provider-cli/client/wallet.md similarity index 94% rename from src/storage-provider-cli/wallet.md rename to src/storage-provider-cli/client/wallet.md index a255b58..08a355c 100644 --- a/src/storage-provider-cli/wallet.md +++ b/src/storage-provider-cli/client/wallet.md @@ -30,7 +30,7 @@ Keys shown on this page are, by default, not secure! Do not use them in producti Generate a new random key to interact with the Polka Storage parachain: ```bash -> polka-storage-provider wallet generate +> polka-storage-provider-client wallet generate Secret phrase: offer payment boost boy manage car asset lock cousin mountain vehicle setup Network ID: substrate Secret seed: 0xfe36ee692552b0ce54de06ce4f5cc152fe2fa808cb40f58c81168bc1237208bb @@ -43,7 +43,7 @@ Secret phrase: offer payment boost boy manage car asset lock cousin mounta The password may be added interactively, using `--password-interactive` flag: ```bash -> polka-storage-provider wallet generate --password-interactive +> polka-storage-provider-client wallet generate --password-interactive Key password: