Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SIMD-0082: Relax Transaction Constraints #82

Closed
173 changes: 173 additions & 0 deletions proposals/0082-relax-transaction-constraints.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
---
simd: '0082'
title: Relax Transaction Constraints
authors:
- Andrew Fitzgerald (Solana Labs)
category: Standard
type: Core
status: Draft
created: 2023-10-30
feature:
---

## Summary

This proposal aims to relax some of the constraints on which individual
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to relax some of the constraints

to clarify, "relax" means "removal"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The constraints still exist in the sense we need to check "can this transaction be executed". They do not exist for the validity of the block.

Right now, those 2 different types of constraints are overlapping. I need to re-write this proposal with @lheeger-jump's suggestion on making it very explicit that these 2 things are different.

transactions can be included in a valid block.
The proposal does not relax the constraints required for a transaction to be
executed.

## Motivation

The current protocol places many constraints on the structure and contents
of blocks; if any constraints are broken, block-validators will mark the block
as invalid.
Many of these constraints are necessary, but some of them are not, and lead to
additional complexity in the protocol.
This proposal aims to relax some of the constraints at the individual
transaction level, in order to simplify the protocol, and give more flexibility
to block-producer and block-validator implementations.

More specifically, this proposal aims to relax the constraints which require
account state in order to determine the validity of a transaction's inclusion.
The reason these constraints are targetted specifically, is that reliance on
account state necessarily requires synchronous execution within the protocol.
This proposal on its' own, will not enable asynchronous execution, but it will
remove one of the barriers to asynchronous execution.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bit confused by "aim" in Summary section and Motivations here. As "aim" seems to yield benefit to block producer ("...transaction can be included..."), which is not part of protocol; while Motivation talks about modifying protocol.

Is it correct yo say this proposal has two parts:

  1. remove listed constraints/requirement of transaction for block producer to include them into block; this part solely in Producer implementation, does not modify protocol;
  2. instead of invalidate entire block at earlist failure of transaction due to named constraints, validator only not execute bad transactions but will continue validate the block. (bad transactions included in ledger). This part of protocol altering.

or maybe I am thinking on wrong track?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think only your 2nd point is included in the proposal. Block-production changse are enabled by the changes in the proposal, but not required beyond potential changes to the fee-colection.

instead of invalidate entire block at earlist failure of transaction due to named constraints, validator only not execute bad transactions but will continue validate the block. (bad transactions included in ledger).

yes. Those bad transactions may end up having limited effect if we are still able to collect fees, or if we decide to make the block-producer cover those fees. They will not be executed though.


Proposal does not suggest or require the block-producer to make any changes, but allows for changes. For several of the constraints, the producer is still incentivized to perform the checks (fee-payer, nonce) because otherwise they will not receive rewards for those transactions. There's freedom in how that is done, and the block-producer could potentially do this in a non-strict fashion.

I'll cover this in the updated prop, but essentially block-validation and consensus really consists of 2 primary questions:

  1. is this block valid?
  2. what is the result of the block?

The changes here are intending to make answering this first question, "is this block valid?", a simpler process that does not require executing the block. This proposal separates these questions logically, but w/ current voting we still require the 2nd question, "what is the result of the block?", be answered synchronously.

## Alternatives Considered

1. Do nothing
- This is the simplest option, as we could leave the protocol as is.
However, this leaves the protocol more complex than it needs to be.
2. Relax all but fee-paying constraint
- This was actually the initial proposal, but it was decided that it would
be better to relax as many constraints as possible, rather than just some.
Any reliance on account state, means the protocol requires synchronous
execution.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify, "it would be better to relax as many constraints as possible" just means that the fee-paying constraint is also removed right? I think you could say that directly here.

3. Additionally, relax the address lookup table resolution constraint
- This was considered, since it is a transaction-level constraint that is
depdendent on account-state. However, due to entry-level and block-level
constraints that rely on the address lookup table resolution, this
constraint cannot easily be relaxed without also relaxing those
constraints.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really understand why there is still a constraint for address lookup tables. If the lookup table is invalid for some reason can't validators just charge the fee payer some fees?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ALT resolution contributes to constraints at higher levels i.e. entry and block.

Entries need to have non-conflicting transactions (without adoption of #83).
Blocks are required to not exceed account write limits (currently 12M CUs).

In both these situations, if we allow ALTs to not resolve I think the behavior becomes very unclear. Do we only consider the static accounts? all resolvable accounts? no accounts?
Not clear to me any of those options is ideal or makes sense. I think we'd like to remove this constraint eventually, but I think it probably makes more sense to do after these complicating entry and block constraints are also removed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, thanks for explaining what the entry and block constraints are, I hadn't considered those. FWIW ALTs were designed to be apend only and the warmup time (currently one slot) is configurable by the runtime and can be increased in the future if it helps bankless leaders manage a cache of ALTs. So I think we could try to move forward with removing this constraint.


## New Terminology

None

## Detailed Design

Specifically, this proposal relaxes constraints that a transaction included in
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This list seems to mix "constraints on transactions which invalidate entries and blocks" and "constraints on transaction execution which invalidate the transaction". The list should be split up or the ones of the latter type should be removed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a very fair comment, as that is something is changing with this proposal. Currently, the list of constraints that invalidate the block is a superset of those which invalidate execution. That will no longer be the case, and I should make that more clear.

a block must:

1. Have a fee-payer with enough funds to pay fees
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This list seems to denote the current status of constraints on transactions. This is useful background, but we need to precisely specify the new constraints on transactions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... in a separate list.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bit confused on what you're suggesting would make the document more clear here; this proposal does not add any new constraints.
The list here is an incomplete list of the constraints on transactions, it is only the constraints that are being removed/relaxed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The list describes what the constraints are now, but does not explicitly list what the relaxed constraints are. Consider that I or another FD dev needs to stare at this and implement the exact rules for what defines a valid transaction.

2. Have a valid nonce account, if specified
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you consider that both normal transactions and durable nonce transactions can only be processed once to be valid? Validators and bankless leaders can use the status cache to avoid including normal transactions more than once, but they need to check that the nonce is valid or not to avoid including durable nonce transactions more than once.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Considered, but I need to rewrite sections of the proposal to be very explicit.

Status cache should still be used to avoid including transactions multiple times. Duplicate inclusion of transactions is still a violation of constraints and leads to the entire block being marked as invalid. This does not depend on account state, which is why I think this should be left in.

With respect to nonces, we want to avoid depdendence on account state, so cannot check the validity of the nonce account for the question of block validity. However, the nonce acount state must still be checked before execution. If the nonce account does not exist or is otherwise invalid, the transaction is not executed and fees are not charged to the fee-payer.
I view this as a similar situation to the fee-payer check; The block-producer is incentivized to check these states. They are free to do that in some non-strict method, but if done incorrectly, they lose out on rewards and the transaction has no/limited on state and have effectively burned their CUs.

There's an ongoing discussion of if fees should be paid by the block-producer should a transaction be unable to pay. So whether or not the producer gets charged fees in these cases is still an open question.

3. Have program accounts that:
1. exist
2. are executable
4. Have writable accounts that are:
1. not executable, unless owned by
`BPFLoaderUpgradeab1e11111111111111111111111`
2. if owned by `BPFLoaderUpgradeab1e11111111111111111111111`,
then `BPFLoaderUpgradeab1e11111111111111111111111` must be included
3. if owned by `Stake11111111111111111111111111111111111111`, then the
current slot must not be within the epoch stakes reward distribution period
5. Have executable `BPFLoaderUpgradeab1e11111111111111111111111` owned accounts
with `UpgradeableLoaderState::Program` state, and derived program data account
that exist
6. Have a call chain depth of 5 or less
7. Have no builtin loader ownership chains
(pending `4UDcAfQ6EcA6bdcadkeHpkarkhZGJ7Bpq7wTAiRMjkoi`)
8. Have total loaded data size which does not exceed
requested_loaded_accounts_data_size_limit
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add links to the feature documents.

(pending `DdLwVYuvDz26JohmgSbA7mjpJFgX5zP2dkp8qsF2C33V`)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried to be as specific as possible in the document itself, but these constraints (3-8) can be summarized as "transaction must be executable"

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

\(^_^)/


The intent with relaxing these constraints is to minimize the amount of
account-state which is required in order to validate a block. This gives more
flexibility to when and how account-state is updated during both block
production and validation.

With these constraints removed, there is still one account-state constraint
at the transaction level: address lookup table resolution.
With this proposal, this constraint is intentionally not relaxed since it is
necessary for validation of entry-level and block-level constraints.
However, if/when those constraints are relaxed, this constraint should be
relaxed as well.

The relaxation of these constraints is only relaxed for transactions being
included in a block. During block validation, if any of these constraints
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this affect transaction simulation? Currently, we get an error when we simulate transactions where the payer does not have enough balance. This will make these kinds of errors indetectable during transaction simulation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what you mean. Transaction simulation is unrelated to block invalidation, and doesn't change with this proposal.
You'd still get the same transaction error which indicates it cannot pay for fees.

are broken, the entire block is marked invalid; with this proposal, the
block is not marked invalid, but the transaction will not be executed.
These constraints must still be satisfied in order for the transaction to be
executed, and have an effect on state. However, there are some new considerations
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe better off being explicit about side-effects on some on-chain states for these constraints-violating transactions. namely, the status cache to avoid repeated inclusion of same transactions.

which must be taken into account, specifically as it relates to fees and block
limits.

### Fee-Paying

Currently, iff a transaction's fee-payer does not have enough funds to pay the
fee, the transaction cannot be included in a block. With this proposal, it is
possible for such a transaction to be included in the block, and there are
three different edge-cases to consider:

1. The fee-payer account does not exist (0 lamports)
2. The fee-payer account does not have enough funds for the entire fee
3. The fee-payer account has enough funds for the fee, but would no longer be
rent-exempt

In case 1, the transaction should simply be ignored for execution, and have no
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should also consider that base fee will eventually change and could be much greater than now.

effect on state.
In case 2, the fee-paying account should be drained of all funds, but have no
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably a good exercise to also enumerate the normal case

other effect on state.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this, Case 2 would never exists, would bad actors fall back to case 1 to quickly build tx with random payer account for DoS attack.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this,

Might have missed a comment this is attached to. With what?

Case 2 would probably not occur if it was a malicious actor, but could happen if some non-strict block-producer made and assumption a fee-payer was good and was not.

In case 3, the fee-paying account should be drained of all funds, with fees
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems a bit harsh, no?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potentially this is too harsh. Alternatively we could just drain them to the rent-exempt level.
Not sure which seems fairer. If submitting a transaction that cannot pay fees, they should be punished imo.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imho, since the commission is paid, the transaction should be executed. Deleting the account is a different event.

being paid to the block-producer, and the remainder of lamports being dropped.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dropped => burnt?


### Block-Limits

Pending the activation of `2ry7ygxiYURULZCrypHhveanvP5tzZ4toRwVp89oCNSj`,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a link.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SIMD should stand for itself and should not have any links, as far as i know.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It at least needs to describe the feature.

validators must validate a block is within block-limits.
With this proposal, some transactions may not be executable, but will still
count towards block-limits.
If these transactions did not count towards block-limits, the validation of
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do they specifically contribute to block limits.

block-limits would require the validator to check whether or not a transaction
is executable, which negates the benefits of this proposal.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the case here is "producer builds a block of 48M cu, but validator only spent less than 48M to execute all transactions (as some of TXs are failed and not executed)", and this should be OK (except the DoS vector - that entire block is unexecutable txs, which can be addressed by making producer pay for these unexecutable txs)


Additionally, f these transactions did not count towards block-limits, a
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Additionally, f these transactions did not count towards block-limits, a
Additionally, if these transactions did not count towards block-limits, a

malicious leader could produce a block with non-executable transactions and
overload the network.

## Impact

- Transactions that would previously be dropped with an error, can now be
included, and even charged fees.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Transactions that would previously be dropped with an error, can now be
included, and even charged fees.
- Transactions that would previously be dropped with an error without paying fees, can now be
included and will be charged fees.

I think "and even charged fees" doesn't sound declarative enough. My understanding is that the crux of this SIMD is that before we dropped txs for a variety of reasons and didn't charge fees and after we will allow invalid txs to be in a block and will charge fees for them.

- Users must be more careful when constructing transactions to ensure they
are executable if they don't want to waste fees
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are there additional tools (or simulator enhancement) needed for transaction submitters ensure their transactions against those relaxed constraints?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simulation doesn't change. It currently would tell you that your transaction had an error. That error doesn't change, it only changes what will happen to a block it may be included in.

For example, if a transaction had an invalid program account, simulation will still return ProgramAccountNotFound error.
If the user simulates, and then still submits that transaction, then they probably deserve to be charged, imo.

- The validity of a block is no longer dependent on intra-block account-state
updates. This is because the only account-state required for transaction
validation is the ALT resolution, which is resolved at the beginning of a slot.
- Block-production could be done asynchronously
- Block-validation could be done without execution, but still relies on the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could be done without execution

Can use some explanations/examples here.

execution of previous blocks.

## Security Considerations
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could any other relaxed constraints, in addition to relaxing "fee-payer requirement", lead to similar dos vector, if block producer build with bad transactions?

Is it necessary to charge transactions failed due to those relaxed constraints by its transaction base fee?


- Removing fee-paying requirement could allow validator clients to produce
blocks that do not pay fees. If not checked by block-producer, this could
allow malicious users to abuse such producers/leaders.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, malicious leaders could create full of garbage blocks, which only contains bad transactions at no cost, burdening all the validations onto other validators, esp, heavy account state lookup.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed with @ryoqun, this attack vector is the main thing that needs to be solved. Maybe this specific problem can be discussed in a separate SIMD since relaxing other transaction constraints is more straightforward.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

made a comment in this chain: #82 (comment)

I don't think there's a DOS that's creating more work than would be possible within current protocol.
I can currently create a block thats full of valid heavy transactions, and then tag in some invalidating transaction at the last tick. All the validators wasted their time executing all those txs, and there's no reward for them since the block is thrown out entirely.

This would change the vector, since they could have all txs that do not pay fees. But, that only makes the execution less intense, since they still have a cost that counts towards block limits.

## Drawbacks

- Any dynamic fees based on block utilization cannot/should not reward the
block-producer directly. Otherwise, this incentivizes block-producers to fill
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any dynamic fees based on block utilization cannot/should not reward the block-producer directly

Humbly not like the sound of a future proposal (eg Dynamic base fee) is already constrained by current design.

But if bock producer cannot include no-paying transactions for free, say producers are liable for adding those bad transactions to block and have to pay for those, then we wouldn't have to worry about this drawback?

blocks with non-fee paying transactions in order to raise fees.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

imo, it is necessary to have someone to pay for ledger space occupied by those failed transactions (that do not update global state machine). There are several ideas, including have Producer pay. Should the discussion of how to do that be part of this SIMD?

- Non-fee paying transactions included in a block will be recorded in long-term
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be a way to attack the cluster and use more space. A malicious leader could create 48M CU / (CU required to load payer account which currently stands at 0) number of transactions. This could be a massive number of transactions, depending on CUs charge that we decide to load the payer account. If each transaction was given the maximum size of 1280 bytes or eventually more if we include ZK transactions, then I can possibly create a huge block burdening the whole cluster's network resources because of the turbine.

I feel like the leader should be somehow punished/slashed for adding transactions where the payer does not have enough balance.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently transactions do have a minimum cost of 1070 CUs = 720 SigVerify + 300 WriteLock.
This corresponds to a maximum of 44776 txs per slot, assuming 48M CU limit.

Additionally, cost has 1 CU per 4 bytes of instruction data. So large ix data txs would cost more than this bare minimum of 1070 CUs as well.
It's still a large number of transactions of 34532 txs per slot.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's worth noting that currently, all direct rewards are going to the block-producer anyway.
If that large number of transactions were valid fee-paying transactions, it would only be more work for other validators than if they could not pay fees. Either way, other nodes are not directly rewarded for the validation/execution.

There is still some discussion to be had around indirect rewards via the burning of SOL in those fees. I think that's potentially a valid reason we may want to force the block-producer into paying fees. Hoping to discuss this more

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I agree, block producers should pay a fee as they are using network resources, storage space to transfer invalid transactions. Malicious node can just start creating blocks with invalid transactions of 1280 bytes using 44.20096 MBs/ block in network and storage.

transaction history. Without any fees being paid, there is no way to reward
long-term storage of these transactions. It is important to note, there is also
currently no way to reward this storage.

## Backwards Compatibility

This proposal is backwards compatible with the current protocol, since it only
relaxes constraints, and does not add any new constraints. All previously valid
blocks would still be valid.