diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md index a9e81e06f..89d90fb14 100644 --- a/book/src/SUMMARY.md +++ b/book/src/SUMMARY.md @@ -11,5 +11,9 @@ - [FPVM Backend](./sdk/fpvm-backend.md) - [Custom Backend](./sdk/custom-backend.md) - [`kona-executor` Extensions](./sdk/exec-ext.md) + - [`kona-derive` Pipeline](./sdk/pipeline/intro.md) + - [Custom Providers](./sdk/pipeline/providers.md) + - [Stage Swapping](./sdk/pipeline/stages.md) + - [Signaling](./sdk/pipeline/signaling.md) - [Glossary](./glossary.md) - [Contributing](./CONTRIBUTING.md) diff --git a/book/src/sdk/pipeline/intro.md b/book/src/sdk/pipeline/intro.md new file mode 100644 index 000000000..b9e32b606 --- /dev/null +++ b/book/src/sdk/pipeline/intro.md @@ -0,0 +1,314 @@ +# The `kona-derive` Derivation Pipeline + +[`kona-derive`][kd] defines an entirely trait-abstracted, `no_std` derivation +pipeline for the OP Stack. It can be used through the [`Pipeline`][p] trait, +which is implemented for the concrete [`DerivationPipeline`][dp] object. + +This document dives into the inner workings of the derivation pipeline, its +stages, and how to build and interface with Kona's pipeline. Other documents +in this section will provide a comprehensive overview of Derivation Pipeline +extensibility including trait-abstracted providers, custom stages, signaling, +and hardfork activation including multiplexed stages. + +- [Swapping out a stage](./stages.md) +- [Defining a custom Provider](./providers.md) +- [Extending Pipeline Signals](./signals.md) +- [Implementing Hardfork Activations](./hardforks.md) + + +## What is a Derivation Pipeline? + +Simply put, an OP Stack Derivation Pipeline transforms data on L1 into L2 +payload attributes that can be executed to produce the canonical L2 block. + +Within a pipeline, there are a set of stages that break up this transformation +further. When composed, these stages operate over the input data, sequentially +producing payload attributes. + +In [`kona-derive`][kd], stages are architected using composition - each sequential +stage owns the previous one, forming a stack. For example, let's define stage A +as the first stage, accepting raw L1 input data, and stage C produces the pipeline +output - payload attributes. Stage B "owns" stage A, and stage C then owns stage B. +Using this example, the [`DerivationPipeline`][dp] type in [`kona-derive`][kd] only +holds stage C, since ownership of the other stages is nested within stage C. + +> [!NOTE] +> +> In a future architecture of the derivation pipeline, stages could be made +> standalone such that communication between stages happens through channels. +> In a multi-threaded, non-fault-proof environment, these stages can then +> run in parallel since stage ownership is decoupled. + + +## Kona's Derivation Pipeline + +The top-level stage in [`kona-derive`][kd] that produces +[`OpAttributesWithParent`][attributes] is the [`AttributesQueue`][attributes-queue]. + +Post-Holocene (the Holocene hardfork), the following stages are composed by +the [`DerivationPipeline`][dp]. +- [`AttributesQueue`][attributes-queue] + - [`BatchProvider`][batch-provider] + - [`BatchStream`][batch-stream] + - [`ChannelReader`][channel-reader] + - [`ChannelProvider`][channel-provider] + - [`FrameQueue`][frame-queue] + - [`L1Retrieval`][retrieval] + - [`L1Traversal`][traversal] + +Notice, from top to bottom, each stage owns the stage nested below it. +Where the [`L1Traversal`][traversal] stage iterates over L1 data, the +[`AttributesQueue`][attributes-queue] stage produces +[`OpAttributesWithParent`][attributes], creating a function that transforms +L1 data into payload attributes. + + +## The [`Pipeline`][p] interface + +Now that we've broken down the stages inside the [`DerivationPipeline`][dp] +type, let's move up another level to break down how the [`DerivationPipeline`][dp] +type functions itself. At the highest level, [`kona-derive`][kd] defines the +interface for working with the pipeline through the [`Pipeline`][p] trait. + +[`Pipeline`][p] provides two core methods. +- `peek() -> Option<&OpAttributesWithParent>` +- `async step() -> StepResult` + +Functionally, a pipeline can be "stepped" on, which attempts to derive +payload attributes from input data. Steps do not guarantee that payload attributes +are produced, they only attempt to advance the stages within the pipeline. + +The `peek()` method provides a way to check if attributes are prepared. +Beyond `peek()` returning `Option::Some(&OpAttributesWithParent)`, the [`Pipeline`][p] +extends the [Iterator][iterator] trait, providing a way to consume the generated payload +attributes. + + +## Constructing a Derivation Pipeline + +[`kona-derive`][kd] provides a [`PipelineBuilder`][builder] to abstract the complexity +of generics away from the downstream consumers. Below we provide an example for using +the [`PipelineBuilder`][builder] to instantiate a [`DerivationPipeline`][dp]. + +```rust +// Imports +use std::sync::Arc; +use op_alloy_protocol::BlockInfo; +use op_alloy_genesis::RollupConfig; +use kona_derive_alloy::prelude::*; + +// Use a default rollup config. +let rollup_config = Arc::new(RollupConfig::default()); + +// Providers are instantiated to with localhost urls (`127.0.0.1`) +let chain_provider = + AlloyChainProvider::new_http("http://127.0.0.1:8545".try_into().unwrap()); +let l2_chain_provider = AlloyL2ChainProvider::new_http( + "http://127.0.0.1:9545".try_into().unwrap(), + rollup_config.clone(), +); +let beacon_client = OnlineBeaconClient::new_http("http://127.0.0.1:5555".into()); +let blob_provider = OnlineBlobProvider::new(beacon_client, None, None); +let blob_provider = OnlineBlobProviderWithFallback::new(blob_provider, None); +let dap_source = + EthereumDataSource::new(chain_provider.clone(), blob_provider, &rollup_config); +let builder = StatefulAttributesBuilder::new( + rollup_config.clone(), + l2_chain_provider.clone(), + chain_provider.clone(), +); + +// This is the starting L1 block for the pipeline. +// +// To get the starting L1 block for a given L2 block, +// use the `AlloyL2ChainProvider::l2_block_info_by_number` +// method to get the `L2BlockInfo.l1_origin`. This l1_origin +// is the origin that can be passed here. +let origin = BlockInfo::default(); + +// Build the pipeline using the `PipelineBuilder`. +// Alternatively, use the `new_online_pipeline` helper +// method provided by the `kona-derive-alloy` crate. +let pipeline = PipelineBuilder::new() + .rollup_config(rollup_config.clone()) + .dap_source(dap_source) + .l2_chain_provider(l2_chain_provider) + .chain_provider(chain_provider) + .builder(builder) + .origin(origin) + .build(); + +assert_eq!(pipeline.rollup_config, rollup_config); +assert_eq!(pipeline.origin(), Some(origin)); +``` + + +## Producing Payload Attributes + +Since the [`Pipeline`][p] trait extends the [`Iterator`][iterator] trait, +producing [`OpAttributesWithParent`][attributes] is as simple as as calling +[`Iterator::next()`][next] method on the [`DerivationPipeline`][dp]. + +Extending the example from above, producing the attributes is shown below. + +```rust +// Import the iterator trait to show where `.next` is sourced. +use core::iter::Iterator; + +// ... +// example from above constructing the pipeline +// ... + +let attributes = pipeline.next(); + +// Since we haven't stepped on the pipeline, +// there shouldn't be any payload attributes prepared. +assert!(attributes.is_none()); +``` + +As demonstrated, the pipeline won't have any payload attributes +without having been "stepped" on. Naively, we can continuously +step on the pipeline until attributes are ready, and then consume them. + +```rust +// Import the iterator trait to show where `.next` is sourced. +use core::iter::Iterator; + +// ... +// example from constructing the pipeline +// ... + +// Continuously step on the pipeline until attributes are prepared. +let l2_safe_head = L2BlockInfo::default(); +loop { + if matches!(pipeline.step(l2_safe_head).await, StepResult::PreparedAttributes) { + // The pipeline has succesfully prepared payload attributes, break the loop. + break; + } +} + +// Since the loop is only broken once attributes are prepared, +// this must be `Option::Some`. +let attributes = pipeline.next().expect("Must contain payload attributes"); + +// The parent of the prepared payload attributes should be +// the l2 safe head that we "stepped on". +assert_eq!(attributes.parent, l2_safe_head); +``` + +Importantly, the above is not sufficient logic to produce payload attributes and drive +the derivation pipeline. There are multiple different `StepResult`s to handle when +stepping on the pipeline, including advancing the origin, re-orgs, and pipeline resets. +In the next section, pipeline resets are outlined. + +For an up-to-date driver that runs the derivation pipeline as part of the fault proof +program, reference kona's [client driver][driver]. + + +## Resets + +When stepping on the [`DerivationPipeline`][dp] produces a reset error, the driver +of the pipeline must perform a reset on the pipeline. This is done by sending a "signal" +through the [`DerivationPipeline`][dp]. Below demonstrates this. + +```rust +// Import the iterator trait to show where `.next` is sourced. +use core::iter::Iterator; + +// ... +// example from constructing the pipeline +// ... + +// Continuously step on the pipeline until attributes are prepared. +let l2_safe_head = L2BlockInfo::default(); +loop { + match pipeline.step(l2_safe_head).await { + StepResult::StepFailed(e) | StepResult::OriginAdvanceErr(e) => { + match e { + PipelineErrorKind::Reset(e) => { + // Get the system config from the provider. + let system_config = l2_chain_provider + .system_config_by_number( + l2_safe_head.block_info.number, + rollup_config.clone(), + ) + .await?; + // Reset the pipeline to the initial L2 safe head and L1 origin. + self.pipeline + .signal( + ResetSignal { + l2_safe_head: l2_safe_head, + l1_origin: pipeline + .origin() + .ok_or_else(|| anyhow!("Missing L1 origin"))?, + system_config: Some(system_config), + } + .signal(), + ) + .await?; + // ... + } + _ => { /* Handling left to the driver */ } + } + } + _ => { /* Handling left to the driver */ } + } +} +``` + + +## Learn More + +[`kona-derive`][kd] is one implementation of the OP Stack derivation pipeline. + +To learn more, it is highly encouraged to read the ["first" derivation pipeline][op-dp] +written in [golang][go]. It is often colloquially referred to as the "reference" +implementation and provides the basis for how much of Kona's derivation pipeline +was built. + + +## Provenance + +> The lore do be bountiful. +> +> - Bard XVIII of the Logic Gates + +The kona project spawned out of the need to build a secondary fault proof for the OP Stack. +Initially, we sought to re-use [magi][magi]'s derivation pipeline, but the ethereum-rust +ecosystem moves quickly and [magi][magi] was behind by a generation of types - using +[ethers-rs] instead of new [alloy][alloy] types. Additionally, [magi][magi]'s derivation +pipeline was not `no_std` compatible - a hard requirement for running a rust fault proof +program on top of the RISCV or MIPS ISAs. + +So, [@clabby][clabby] and [@refcell][refcell] stood up [kona][kona] in a few months. + + + + +[driver]: https://github.com/anton-rs/kona/blob/main/bin/client/src/l1/driver.rs#L74 +[next]: https://doc.rust-lang.org/nightly/core/iter/trait.Iterator.html#tymethod.next +[builder]: https://docs.rs/kona-derive/latest/kona_derive/pipeline/struct.PipelineBuilder.html +[alloy]: https://github.com/alloy-rs/alloy +[ethers-rs]: https://github.com/gakonst/ethers-rs +[kona]: https://github.com/anton-rs/kona +[clabby]: https://github.com/clabby +[refcell]: https://github.com/refcell +[go]: https://go.dev/ +[magi]: https://github.com/a16z/magi +[kd]: https://crates.io/crates/kona-derive +[iterator]: https://doc.rust-lang.org/nightly/core/iter/trait.Iterator.html +[p]: https://docs.rs/kona-derive/latest/kona_derive/traits/trait.Pipeline.html +[op-dp]: https://github.com/ethereum-optimism/optimism/tree/develop/op-node/rollup/derive +[dp]: https://docs.rs/kona-derive/latest/kona_derive/pipeline/struct.DerivationPipeline.html +[attributes]: https://docs.rs/op-alloy-rpc-types-engine/latest/op_alloy_rpc_types_engine/struct.OpAttributesWithParent.html + + + +[attributes-queue]: https://docs.rs/kona-derive/latest/kona_derive/stages/struct.AttributesQueue.html +[batch-provider]: https://docs.rs/kona-derive/latest/kona_derive/stages/struct.BatchProvider.html +[batch-stream]: https://docs.rs/kona-derive/latest/kona_derive/stages/struct.BatchStream.html +[channel-reader]: https://docs.rs/kona-derive/latest/kona_derive/stages/struct.ChannelReader.html +[channel-provider]: https://docs.rs/kona-derive/latest/kona_derive/stages/struct.ChannelProvider.html +[frame-queue]: https://docs.rs/kona-derive/latest/kona_derive/stages/struct.FrameQueue.html +[retrieval]: https://docs.rs/kona-derive/latest/kona_derive/stages/struct.L1Retrieval.html +[traversal]: https://docs.rs/kona-derive/latest/kona_derive/stages/struct.L1Traversal.html diff --git a/book/src/sdk/pipeline/providers.md b/book/src/sdk/pipeline/providers.md new file mode 100644 index 000000000..bb2de1c7d --- /dev/null +++ b/book/src/sdk/pipeline/providers.md @@ -0,0 +1,113 @@ +# Trait-abstracted Providers + +Kona's derivation pipeline pulls in data from sources that are trait +abstracted so the pipeline can be generic over various data sources. +Note, "data sources" is used interchangeably with "trait-abstracted +providers" for the purpose of this document. + +The key traits required for the pipeline are the following. + +- [`ChainProvider`][chain-provider] +- [`L2ChainProvider`][l2-chain-provider] +- [`DataAvailabilityProvider`][dap] + +The [`kona-derive-alloy`][kda] crate provides `std` implementations +of these traits using [Alloy][alloy]'s `reqwest`-backed providers. + +## Provider Usage + +Although trait-abstracted Providers are used throughout the pipeline and +its stages, the [`PipelineBuilder`][builder] makes constructing the pipeline +generic over the providers. An example is shown below, where the three +required trait implementations are the providers stubbed with `todo!()`. + +```rust +use std::sync::Arc; +use op_alloy_genesis::RollupConfig; +use kona_derive::pipeline::PipelineBuilder; +use kona_derive::attributes::StatefulAttributesBuilder; + +// The rollup config for your chain. +let cfg = Arc::new(RollupConfig::default()); + +// Must implement the `ChainProvider` trait. +let chain_provider = todo!("your chain provider"); + +// Must implement the `L2ChainProvider` trait. +let l2_chain_provider = todo!("your l2 chain provider"); + +// Must implement the `DataAvailabilityProvider` trait. +let dap = todo!("your data availability provider"); + +// Generic over the providers. +let attributes = StatefulAttributesBuilder::new( + cfg.clone(), + l2_chain_provider.clone(), + chain_provider.clone(), +); + +// Construct a new derivation pipeline. +let pipeline = PipelineBuilder::new() + .rollup_config(cfg) + .dap_source(dap) + .l2_chain_provider(l2_chain_provider) + .chain_provider(chain_provider) + .builder(attributes) + .origin(BlockInfo::default()) + .build(); +``` + +## Implementing a Custom Data Availability Provider + +> Notice +> +> The only required method for the [`DataAvailabilityProvider`][dap] +> trait is the [`open_data`][open-data] method. + +```rust +use async_trait::async_trait; +use alloy_primitives::Bytes; +use op_alloy_protocol::BlockInfo; +use kona_derive::traits::DataAvailabilityProvider; +use kona_derive::errors::PipelineResult; + +/// ExampleAvail +/// +/// An example implementation of the `DataAvailabilityProvider` trait. +#[derive(Debug)] +pub struct ExampleAvail { + // Place your data in here +} + +#[async_trait] +impl AsyncIterator for ExampleDataIterator { + type Item = Bytes; + + async fn next(&mut self) -> PipelineResult { + todo!("return the next bytes") + } +} + + +#[async_trait] +impl DataAvailabilityProvider for ExampleAvail { + type Item = Bytes; + type DataIter = ExampleDataIterator; + + async fn open_data(&self, block_ref: &BlockInfo) -> PipelineResult { + todo!("return an AsyncIterator implementation here") + } +} +``` + + + + +[dap]: https://docs.rs/kona-derive/latest/kona_derive/traits/trait.DataAvailabilityProvider.html +[open-data]: https://docs.rs/kona-derive/latest/kona_derive/traits/trait.DataAvailabilityProvider.html#tymethod.open_data +[builder]: https://docs.rs/kona-derive/latest/kona_derive/pipeline/struct.PipelineBuilder.html +[alloy]: https://github.com/alloy-rs/alloy +[kda]: https://crates.io/crates/kona-derive-alloy +[chain-provider]: https://docs.rs/kona-derive/latest/kona_derive/traits/trait.ChainProvider.html +[l2-chain-provider]: https://docs.rs/kona-derive/latest/kona_derive/traits/trait.L2ChainProvider.html +[dap]: https://docs.rs/kona-derive/latest/kona_derive/traits/trait.DataAvailabilityProvider.html diff --git a/book/src/sdk/pipeline/signaling.md b/book/src/sdk/pipeline/signaling.md new file mode 100644 index 000000000..eaa367df7 --- /dev/null +++ b/book/src/sdk/pipeline/signaling.md @@ -0,0 +1,156 @@ +# Signals + +Understanding signals first require a more in-depth review of the result +returned by stepping on the derivation pipeline. + + +## The [`StepResult`][step-result] + +As briefly outlined in the [intro](./intro.md), stepping on the derivation +pipeline returns a [`StepResult`][step-result]. Step results provide a +an extensible way for pipeline stages to signal different results to the +pipeline driver. The variants of [`StepResult`][step-result] and what they +signal include the following. + +- `StepResult::PreparedAttributes` - signals that payload attributes are + ready to be be consumed by the pipeline driver. +- `StepResult::AdvancedOrigin` - signals that the pipeline has derived all + payload attributes for the given L1 block, and the origin of the pipeline + was advanced to the next canonical L1 block. +- `StepResult::OriginAdvanceErr(_)` - The driver failed to advance the + origin of pipeline. +- `StepResult::StepFailed(_)` - The step failed. + +No action is needed when the prepared attributes step result is received. +The pipeline driver may chose to consume the payload attributes how it +wishes. Likewise, `StepResult::AdvancedOrigin` simply notifies the driver +that the pipeline advanced its origin - the driver may continue stepping +on the pipeline. Now, it becomes more involved with the remaining two +variants of [`StepResult`][step-result]. + +When either `StepResult::OriginAdvanceErr(_)` or `StepResult::StepFailed(_)` +are received, the pipeline driver needs to introspect the error within these +variants. Depending on the [`PipelineErrorKind`][error-kind], the driver may +need to send a "signal" down through the pipeline. + +The next section goes over pipeline signals by looking at the variants of +the [`PipelineErrorKind`][error-kind] and the driver's response. + + +## [`PipelineErrorKind`][error-kind] + +There are three variants of the [`PipelineErrorKind`][error-kind], each +groups the inner error based on severity (or how they should be handled). + +- `PipelineErrorKind::Temporary` - This is an error that's expected, and + is temporary. For example, not all channel data has been posted to L1 + so the pipeline doesn't have enough data yet to continue deriving + payload attributes. +- `PipelineErrorKind::Critical` - This is an unexpected error that breaks + the derivation pipeline. It should cause the driver to error since this + is behavior that is breaking the derivation of payload attributes. +- `PipelineErrorKind::Reset` - When this is received, it effectively + requests that the driver perform some action on the pipeline. Kona + uses message passing so the driver can send a [`Signal`][signal] down + the pipeline with whatever action that needs to be performed. By + allowing both the driver and individual pipeline stages to define their + own behaviour around signals, they become very extensible. More on this + in [a later section](#extending-the-signal-type). + + +## The [`Signal`][signal] Type + +Continuing from the [`PipelineErrorKind`][error-kind], when the driver +receives a `PipelineErrorKind::Reset`, it needs to send a signal down +through the pipeline. + +Prior to the Holocene hardfork, the pipeline only needed to be reset +when the reset pipeline error was received. Holocene activation rules +changed this to require Holocene-specific activation logic internal to +the pipeline stages. The way kona's driver handles this activation is +by sending a new `ActivationSignal` if the `PipelineErrorKind::Reset` +type is a `ResetError::HoloceneActivation`. Otherwise, it will send the +`ResetSignal`. + +The last of the three [`Signal`][signal] variants is the `FlushChannel` +signal. Similar to `ActivationSignal`, the flush channel signal is logic +introduced post-Holocene. When the driver fails to execute payload +attributes and Holocene is active, a `FlushChannel` signal needs to +forwards invalidate the associated batch and channel, and the block +is replaced with a deposit-only block. + + +## Extending the Signal Type + +To extend the [`Signal`][signal] type, all that is needed is to introduce +a new variant to the [`Signal`][signal] enum. + +Once the variant is added, the segments where signals are handled need to +be updated. Anywhere the [`SignalReceiver`][receiver] trait is +implemented, handling needs to be updated for the new signal variant. Most +notably, this is on the top-level [`DerivationPipeline`][dp] type, as well +as all [the pipeline stages][stages]. + +#### An Example + +Let's create a new [`Signal`][signal] variant that updates the `RollupConfig` +in the [`L1Traversal`][traversal] stage. Let's call it `SetConfig`. +The [`signal`][signal] type would look like the following with this new +variant. + +```rust +/// A signal to send to the pipeline. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[allow(clippy::large_enum_variant)] +pub enum Signal { + /// Reset the pipeline. + Reset(ResetSignal), + /// Hardfork Activation. + Activation(ActivationSignal), + /// Flush the currently active channel. + FlushChannel, + /// Updates the rollup config in the L1Traversal stage. + UpdateConfig(ConfigUpdateSignal), +} + +/// A signal that updates the `RollupConfig`. +#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)] +pub struct ConfigUpdateSignal(Arc); +``` + +Next, all handling of the [`Signal`][signal] type needs to be updated for +the new `UpdateConfig` variant. For the sake of this example, we'll just +focus on updating the [`L1Traversal`][traversal] stage. + +```rust +#[async_trait] +impl SignalReceiver for L1Traversal { + async fn signal(&mut self, signal: Signal) -> PipelineResult<()> { + match signal { + Signal::Reset(ResetSignal { l1_origin, system_config, .. }) | + Signal::Activation(ActivationSignal { l1_origin, system_config, .. }) => { + self.block = Some(l1_origin); + self.done = false; + self.system_config = system_config.expect("System config must be provided."); + } + Signal::UpdateConfig(inner) => { + self.rollup_config = Arc::clone(&inner.0); + } + _ => {} + } + + Ok(()) + } +} +``` + + + + +[traversal]: https://docs.rs/kona-derive/latest/kona_derive/stages/struct.L1Traversal.html +[dp]: https://docs.rs/kona-derive/latest/kona_derive/pipeline/struct.DerivationPipeline.html +[stages]: https://docs.rs/kona-derive/latest/kona_derive/stages/index.html +[receiver]: https://docs.rs/kona-derive/latest/kona_derive/traits/trait.SignalReceiver.html +[signal]: https://docs.rs/kona-derive/latest/kona_derive/traits/enum.Signal.html +[error-kind]: https://docs.rs/kona-derive/latest/kona_derive/errors/enum.PipelineErrorKind.html +[step-result]: https://docs.rs/kona-derive/latest/kona_derive/traits/enum.StepResult.html diff --git a/book/src/sdk/pipeline/stages.md b/book/src/sdk/pipeline/stages.md new file mode 100644 index 000000000..97b1b7517 --- /dev/null +++ b/book/src/sdk/pipeline/stages.md @@ -0,0 +1,113 @@ +# Swapping out a Stage + +In the [introduction to the derivation pipeline][intro], the derivation pipeline +is broken down to demonstrate the composition of stages, forming the transformation +function from L1 data into L2 payload attributes. + +What makes kona's derivation pipeline extensible is that stages are composed using +trait-abstraction. That is, each successive stage composes the previous stage as +a generic. As such as long as a stage satisfies two rules, it can be swapped into +the pipeline seamlessly. +1. The stage implements the trait required by the next stage. +2. The stage uses the same trait for the previous stage as the + current stage to be swapped out. + +Below provides a concrete example, swapping out the `L1Retrieval` stage. + +## Example + +In the current, post-Holocene hardfork [`DerivationPipeline`][dp], the bottom three +stages of the pipeline are as follows (from top down). + +- [`FrameQueue`][frame-queue] +- [`L1Retrieval`][retrieval] +- [`L1Traversal`][traversal] + +In this set of stages, the [`L1Traversal`][traversal] stage sits at the bottom. +It implements the [`L1Retrieval`][retrieval] trait called the +[`L1RetrievalProvider`][retrieval-provider]. This provides generic methods that +allow the [`L1Retrieval`][retrieval] stage to call those methods on the generic +previous stage that implements this provider trait. + +As we go up a level, the same trait abstraction occurs. The [`L1Retrieval`][retrieval] +stage implements the provider trait that the [`FrameQueue`][frame-queue] stage requires. +This trait is the [`FrameQueueProvider`][frame-queue-provider]. + +Now that we understand the trait abstractions, let's swap out the +[`L1Retrieval`][retrieval] stage for a custom `DapRetrieval` stage. + +```rust +// ... +// imports +// ... + +// We use the same "L1RetrievalProvider" trait here +// in order to seamlessly use the `L1Traversal` + +/// DapRetrieval stage +#[derive(Debug)] +pub struct DapRetrieval

+where + P: L1RetrievalProvider + OriginAdvancer + OriginProvider + SignalReceiver, +{ + /// The previous stage in the pipeline. + pub prev: P, + provider: YourDataAvailabilityProvider, + data: Option, +} + +#[async_trait] +impl

FrameQueueProvider for DapRetrieval

+where + P: L1RetrievalProvider + OriginAdvancer + OriginProvider + SignalReceiver + Send, +{ + type Item = Bytes; + + async fn next_data(&mut self) -> PipelineResult { + if self.data.is_none() { + let next = self + .prev + .next_l1_block() + .await? // SAFETY: This question mark bubbles up the Eof error. + .ok_or(PipelineError::MissingL1Data.temp())?; + self.data = Some(self.provider.get_data(&next).await?); + } + + match self.data.as_mut().expect("Cannot be None").next().await { + Ok(data) => Ok(data), + Err(e) => { + if let PipelineErrorKind::Temporary(PipelineError::Eof) = e { + self.data = None; + } + Err(e) + } + } + } +} + +// ... +// impl OriginAdvancer for DapRetrieval +// impl OriginProvider for DapRetrieval +// impl SignalReceiver for DapRetrieval +// .. +``` + +Notice, the `L1RetrievalProvider` is used as a trait bound so the +[`L1Traversal`][traversal] stage can be used seamlessly as the "prev" stage in the pipeline. +Concretely, an instantiation of the `DapRetrieval` stage could be the following. + +``` +DapRetrieval> +``` + + + + +[intro]: ./intro.md +[dp]: https://docs.rs/kona-derive/latest/kona_derive/pipeline/struct.DerivationPipeline.html +[retrieval-provider]: https://docs.rs/kona-derive/latest/kona_derive/stages/trait.L1RetrievalProvider.html +[frame-queue-provider]: https://docs.rs/kona-derive/latest/kona_derive/stages/trait.FrameQueueProvider.html + +[frame-queue]: https://docs.rs/kona-derive/latest/kona_derive/stages/struct.FrameQueue.html +[retrieval]: https://docs.rs/kona-derive/latest/kona_derive/stages/struct.L1Retrieval.html +[traversal]: https://docs.rs/kona-derive/latest/kona_derive/stages/struct.L1Traversal.html