feat(benchmark): create new `benchmark_test` test type

[LouisTsai-Csie]

Background

Following discussions around general benchmark refactoring and the transaction gas limit cap issue, the ZkEVM team, Nethermind team (for gas benchmarking purpose),along with other teams, agreed that using the blockchain_test format is sufficient. This aligns with zkEVM’s goal of proving entire blocks rather than individual transactions (as in state_test).

However, most existing test cases are currently in the state_test format, which requires conversion using the reth deserialization tools. Additionally, EIP-7825 has been introduced in Fusaka, which enforces a maximum transaction gas limit of 2^24. This requires us to rewrite a lot of benchmark tests. Since the test must be split into multiple parts and use the blockchain_test format when the transaction gas exceeds the cap.

Proposed Solution

Introduce a new test type: benchmark_test, designed specifically for benchmark test cases. This format would support:

• Act as a wrapper and only generates the blockchain_test format as output
• High compatibility with both blockchain_test and state_test formats
• Support EIP-7825 for transaction limit cap

Usage Scenarios

• For existing blockchain_test cases: If the blocks field is present, benchmark_test will execute using the blockchain_test format:

blockchain_test(
    genesis_environment=Environment(),
    pre=pre,
    post=post_state,
    blocks=[Block(txs=txs)],
    exclude_full_post_state_in_output=True,
)

->

benchmark_test(
    genesis_environment=Environment(),
    pre=pre,
    post=post_state,
    blocks=[Block(txs=txs)],
    exclude_full_post_state_in_output=True,
)

• For existing state_test input: If only a single transaction is provided, benchmark_test will convert it into a blockchain_test:

state_test(
    env=Environment(),
    pre=pre,
    post={},
    tx=tx,
)

->

benchmark_test(
    env=Environment(),
    pre=pre,
    post={},
    tx=tx,
)

In both scenarios, benchmark_test will:

• Verify the gas limit specified in the transaction.
• If EIP-7825 is active, compare the transaction gas limit against the 2**24 cap:
• If the limit exceeds the cap, split the test into multiple parts accordingly.
• If EIP-7825 is not active, proceed with the original gas limit.

[spencer-tb]

Thanks for adding the issue. Dan briefly spoke to the geth team and I think they wanted to keep state_test. We could still remove it nonetheless. We should still add the following where bencmark_test is simply a wrapper around state_test/blockchain_test.

For opcodes/precompiles added in a fork <Osaka, the benchmark test wrapper would select a state test. For those >= Osaka the test would select the blockchain test variant. I think we could also write in such way where most of the logic is handled by the wrapper, so if we wanted to remove state test completely in the future it would be simple isolated change!

[marioevz]

I like the idea!

One caveat is that we would need to define benchmark_test and benchmark_state_test as different specs because the fixture formats they produce are statically defined, and have to be defined before the test function is executed. See example here.

Another comment is that, taking into account this #1895 (comment), we have to modify the test function to receive the env from the fixture, and also pass along the gas_benchmark_value from #1895 (maybe just so that, internally in benchmark_test, as a sanity check the last block is verified to not use more gas than expected?)

def test_some_benchmark(benchmark_test: BenchmarkTestFiller, gas_benchmark_value: int, env: Environment):
    ... # Test logic
    benchmark_test(
        env=env,
        pre=pre,
        post=post_state,
        gas_benchmark_value=gas_benchmark_value,
        setup_blocks=[Block(txs=setup_txs)],
        benchmark_block=Block(txs=attack_txs),
        exclude_full_post_state_in_output=True,
    )

For the state test, we can simplify and use the code_loop_precompile_call(setup_bytecode, attack_bytecode, fork) template it's normally used in benchmark state tests so benchmark_state_test generates a lot of the boilerplate code internally, more automatically and more consistently:

def test_some_state_benchmark(benchmark_state_test: BenchmarkStateTestFiller, gas_benchmark_value: int, env: Environment):
    ... # Test logic
    benchmark_state_test(
        env=env,
        pre=pre,
        post=post_state,
        gas_benchmark_value=gas_benchmark_value,
        setup_bytecode=setup_bytecode,
        attack_bytecode=attack_bytecode,
    )

Just an idea, there might be something else that works better.

[marioevz]

One the last comment on benchmark_state_test I just thought about: taking into account EIP-7825, the test format, when generating a blockchain test, should cap the tx limit and fill the block with multiple of these txs automatically.

[LouisTsai-Csie]

After reviewing the state_test again, based on my current understanding, it also generates a blockchain test fixture. Does that mean we might not need to remove state_test after all?

[spencer-tb]

After reviewing the state_test again, based on my current understanding, it also generates a blockchain test fixture. Does that mean we might not need to remove state_test after all?

Exactly so for context with fill currently the following gets generated from the defined test:

• state_test() -> state test fixture, blockchain test fixture (just the tx defined in state test wrapped in a block), blockchain engine test fixture (the block but in the engine new payload format).
• blockchain_test() -> blockchain test fixture, blockchain engine test fixture (block or blocks in the engine new payload format).

Regarding removing state test, for me the idea came to mind for simplicity. If no one was using it for benchmarking then it made sense to remove it and just use blockchain test. As some teams including geth/evmone like the state test format its worth keeping for now imo. Maybe there is also added benefit to help clients optimize other code paths for opcodes/precompiles < Osaka.

[danceratopz]

@LouisTsai-Csie @marioevz This new test type should additionally create a "benchmark contract code fixture" (or something with a better name :) ) that is required by spamoor, I create an issue here so we don't forget it:

• feat(fill,benchmark): add new contract-only fixture format for spamoor execution-specs#1569

[LouisTsai-Csie]

For the state test, we can simplify and use the code_loop_precompile_call(setup_bytecode, attack_bytecode, fork) template it's normally used in benchmark state tests so benchmark_state_test generates a lot of the boilerplate code internally, more automatically and more consistently:

This is a good idea. However, after reviewing the test cases, I think this approach can only be applied to a limited subset of tests.

In general, there are two main scenarios:

• Looping Execution Pattern: This includes setup bytecode and attack bytecode. The execution flow is: setup → JUMPDEST → attack → JUMP, which loops back to the origin and starts execution again.

• Maximal Stack Size Coverage: This involves two contracts. Contract A contains a setup bytecode followed by repeated attack bytecode instructions until the EVM stack is filled with 1024 items. Contract B implements a STATICCALL loop: its code is filled with call operations up to the maximum allowed code size, then it jumps back to the beginning and executes again. The target of the static calls is Contract A.

We could consider creating two class objects, LoopOverCode and CoverMaxStakcSize (we need better naming), to automatically generate the necessary bytecode for each scenario. These wouldn’t be implemented in benchmark_state_test; instead, we pass the generated code into the benchmark_state_test wrapper, which would then handle the transaction gas limit logic automatically.

cc @marioevz

[marioevz]

We could consider creating two class objects, LoopOverCode and CoverMaxCodeSize, to automatically generate the necessary bytecode for each scenario.

Yes, I think this is spot on. We could introduce an abstract base class (e.g., BenchmarkCodeGenerator) responsible for generating a single transaction up to a given gas limit.

Both LoopOverCode and CoverMaxCodeSize would inherit from this base class. They would be instantiated with their specific components (e.g., setup, attack, etc.) and implement the abstract method to build transactions, generate_transaction, that would use these inputs to generate a single new transaction every time called.

The abstract method would require parameters:

• pre: we need it to deploy the contracts and the account actually sending the transaction.
• gas_limit: the maximum gas allowed for the transaction.

benchmark_test would take instances of LoopOverCode or CoverMaxCodeSize and internally call this transaction-generation method, potentially multiple times in the case of Osaka where we cannot have a transaction with a gas limit higher than 16M (e.g. in Osaka we would call generate_transaction with 16M or less multiple times).

Something like this:

@pytest.mark.valid_from("Cancun")
@pytest.mark.parametrize("mem_size", [0, 1, 1_000, 100_000, 1_000_000])
def test_benchmark_something(
    benchmark_test: BenchmarkTestFiller,
    pre: Alloc,
    fork: Fork,
    #  gas_benchmark_value: int,  # No longer needed IMO, this can be grabbed internally by `benchmark_test`
):
    setup = ...
    attack = ...
    code_generator = CoverMaxCodeSize(setup=setup, attack=attack)
    benchmark_test(
        pre=pre,
        code_generator=code_generator,
    )

Btw I think we should focus first only on benchmark_test and not benchmark_state_test since this one could become obsolete after Osaka (maybe not but benchmark_test we'll definitely need).

[LouisTsai-Csie]

Closed since PR #1945 is done