Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add functions for multithreaded validation to the Rust side. #541

Open
wants to merge 122 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
122 commits
Select commit Hold shift + click to select a range
9628d74
port errors and npc_result to rust
matt-o-how Apr 25, 2024
b1d414e
add function bones for filling in
matt-o-how Apr 25, 2024
8ec9f22
add BlockGenerator and simple_solution_generator
matt-o-how Apr 30, 2024
cbcd4e2
get_flags_for_height_and_constants
matt-o-how Apr 30, 2024
3c8fa30
add get_name_puzzle_conditions
matt-o-how Apr 30, 2024
3a7d6d5
type fixes
matt-o-how May 1, 2024
e02aa6c
more type fixes
matt-o-how May 2, 2024
f6896c9
pkm_pairs
matt-o-how May 6, 2024
2ce2d7b
type fixes
matt-o-how May 6, 2024
3da2738
use blscache
matt-o-how May 15, 2024
844751d
remove references to npcresult
matt-o-how May 16, 2024
8f8eace
minor fixes and improvements
matt-o-how May 20, 2024
68ec039
add syncing toggle
matt-o-how May 20, 2024
424d6e4
makes errs pub
matt-o-how May 22, 2024
07cf8b4
commit before rebase
matt-o-how May 23, 2024
4f6afb8
move to use ConsensusConstants and cargo fixes
matt-o-how May 24, 2024
f3be475
more cargo fixes
matt-o-how May 24, 2024
e24026c
even more cargo fixes
matt-o-how May 24, 2024
e4d518e
continued cargo work
matt-o-how May 24, 2024
46ee479
more compilation error fixes
matt-o-how May 28, 2024
1fe83b9
fix make_aggsig_final_message
matt-o-how May 28, 2024
cecee94
remove unnecessary clones
matt-o-how May 28, 2024
2c12d92
yet more cargo fixes
matt-o-how May 29, 2024
7a9f8ea
handle other opcode cases
matt-o-how May 29, 2024
d18120c
fix simple_solution_generator
matt-o-how May 29, 2024
4949012
fmt
matt-o-how May 29, 2024
088e834
test runs!
matt-o-how May 30, 2024
b91f098
pass test
matt-o-how May 30, 2024
dcbf2e3
fmt
matt-o-how May 30, 2024
6467689
clippy fixes
matt-o-how May 30, 2024
c754324
pass constants as reference
matt-o-how May 30, 2024
0fb2322
remove unnecessary clones
matt-o-how May 30, 2024
a16b487
use ? instead of match
matt-o-how May 30, 2024
6d2a79d
another fmt
matt-o-how May 30, 2024
4a783f0
small arvid suggestions
matt-o-how May 31, 2024
311c860
remove BlockGenerator from get_name_puzzle_conditions
matt-o-how May 31, 2024
d023c2a
reduce clones in loop
matt-o-how May 31, 2024
948ea39
save an allocation
matt-o-how May 31, 2024
8ff50f0
unused import
matt-o-how May 31, 2024
9c80a65
remove block validation function stubs
matt-o-how Jun 3, 2024
c5053fc
remove calls to run_block_generator in get_npc
matt-o-how Jun 3, 2024
c3b5d92
no longer create a generator program
matt-o-how Jun 4, 2024
30003a7
clippy fixes
matt-o-how Jun 4, 2024
42e4b09
use consenusconstants for additional_data instead of hashmap
matt-o-how Jun 4, 2024
4399e8a
switch pkm_pairs to return an iter
matt-o-how Jun 4, 2024
c01bcbe
optimise via inlining pkm_pairs
matt-o-how Jun 5, 2024
24ca094
fmt + clippy
matt-o-how Jun 5, 2024
42b29b0
add unsafes to pkm_iter and add test for it
matt-o-how Jun 6, 2024
fd4a8d8
remove further clones inside iter
matt-o-how Jun 7, 2024
2607c70
add aggsig_me test
matt-o-how Jun 7, 2024
20cb94d
Update crates/chia-consensus/src/multiprocess_validation.rs
matt-o-how Jun 7, 2024
6a54ea0
update get_flags and fix npcresult unwrap
matt-o-how Jun 7, 2024
2a2f354
Use only one aggregate_verify with a (pk, msg) iter
matt-o-how Jun 7, 2024
a0e35df
fmt
matt-o-how Jun 7, 2024
2b90681
remove useless vec cast
matt-o-how Jun 7, 2024
0b5064f
remove syncing, return additions
matt-o-how Jun 10, 2024
d96dab2
fix cache benchmark
matt-o-how Jun 10, 2024
3775939
add aggsig parent puzzle test and remove syncing and fix cache benchmark
matt-o-how Jun 10, 2024
a4ae257
test_validate_aggsig_parent_amount
matt-o-how Jun 10, 2024
d7a2d8d
add test_validate_aggsig_puzzle_amount
matt-o-how Jun 10, 2024
3845a2c
add pyfunction for validate_clvm_and_signature
matt-o-how Jun 11, 2024
af104f6
OwnedSpendBundleConditions never fails now
matt-o-how Jun 17, 2024
b5cf9c2
remove NO_RELATIVE_CONDITIONS_ON_EPHEMERAL flag as it is now uncondit…
matt-o-how Jun 17, 2024
dc53e40
cargo and fmt fixes
matt-o-how Jun 17, 2024
9f69b59
reduce allocations in make_aggsig_final_message
matt-o-how Jun 17, 2024
c0886f7
use hex!
matt-o-how Jun 17, 2024
d453405
add cache.update() and return additions as vec
matt-o-how Jun 18, 2024
e11152b
clippy and fmt
matt-o-how Jun 18, 2024
bea5393
Update crates/chia-consensus/src/multiprocess_validation.rs
matt-o-how Jun 18, 2024
62d1e89
delete BlockGenerator struct
matt-o-how Jun 18, 2024
c6a022d
apply some of arvid's suggestions
matt-o-how Jun 18, 2024
fc1d307
add test to u64_to_bytes
matt-o-how Jun 18, 2024
20c2775
implement more of arvid's comments
matt-o-how Jun 18, 2024
38a08ff
unused import
matt-o-how Jun 18, 2024
33cbfbc
remove unused pyo3 import
matt-o-how Jun 24, 2024
7e84370
add pytest and type stubs
matt-o-how Jun 25, 2024
1654ebf
if no cache passed in, then create one
matt-o-how Jul 8, 2024
d4d0c23
explicitly call conversion
matt-o-how Jul 8, 2024
bfc24ab
cargo fmt
matt-o-how Jul 9, 2024
e7297da
add costing test
matt-o-how Jul 9, 2024
6ed3814
fix DedupFlags not setting properly
matt-o-how Jul 9, 2024
ff30619
return cache additions as bytes
matt-o-how Jul 9, 2024
61697a2
return badaggsig instead of generic badspendbundle
matt-o-how Jul 10, 2024
3280753
create pybinding for get_name_puzzle_conditions
matt-o-how Jul 15, 2024
6230278
clippy
matt-o-how Jul 16, 2024
b90a72c
add type stubs for get_name_puzzle_conditions
matt-o-how Jul 17, 2024
4fb81b5
pass whole GTElement and update with iter
matt-o-how Jul 18, 2024
83d7c5a
Merge branch 'multithread_validation' of https://github.com/Chia-Netw…
matt-o-how Jul 18, 2024
e636637
python items() returns GTElement
matt-o-how Jul 18, 2024
dc46758
use chia_rs values
matt-o-how Jul 19, 2024
e625725
use ARC pointer and don't clone test list
matt-o-how Jul 22, 2024
3359fa9
return python object for GTElement instead of bytes when calling BLSC…
matt-o-how Jul 22, 2024
3861339
black test_blscache.py
matt-o-how Jul 22, 2024
211b63a
stubs fix
matt-o-how Jul 22, 2024
574521b
optimise tests
matt-o-how Jul 23, 2024
451302e
Merge branch 'main' into multithread_validation
matt-o-how Jul 24, 2024
029b120
fix removal of softfork flag
matt-o-how Jul 25, 2024
8768baf
add Additions and ValidationResult types for Clippy's sake
matt-o-how Jul 25, 2024
54318db
Add interior mutability to BLSCache
Rigidity Jul 22, 2024
facd7b5
fix final pytest
matt-o-how Jul 23, 2024
a91aa35
black tests
matt-o-how Jul 23, 2024
3438e6e
Merge pull request #623 from Chia-Network/cache-interior-mutability
matt-o-how Jul 26, 2024
3825fc3
remove commented out code
matt-o-how Jul 26, 2024
abc0366
fmt
matt-o-how Jul 26, 2024
d8192cc
remove .collect() call from validate_clvm_and_signature
matt-o-how Jul 26, 2024
cd48403
remove pre-validate spendbundle
matt-o-how Jul 29, 2024
3b14011
move Arc import to tests mod
matt-o-how Jul 29, 2024
7565924
rename get_name_puzzle_conditions to get_conditions_from_spendbundle
matt-o-how Jul 29, 2024
20f95ce
rename multithread_validation to spend_bundle_validation
matt-o-how Jul 29, 2024
d4fb13f
remove mempool mode bool
matt-o-how Jul 29, 2024
44b5dfc
expose get_flags_for_height_and_constants
matt-o-how Jul 29, 2024
bd307b6
move large spendbundle test to a clsp.hex file
matt-o-how Jul 29, 2024
09e4a8a
add regression test to make_aggsig_final_message
matt-o-how Jul 29, 2024
ec77718
clippy fixes
matt-o-how Jul 29, 2024
fced1e3
rename npc_result to spendbundle_conditions
matt-o-how Jul 30, 2024
47c23d7
remove need for mut in flags
matt-o-how Jul 30, 2024
019b2f1
Revert "rename get_name_puzzle_conditions to get_conditions_from_spen…
matt-o-how Jul 30, 2024
48c706a
re-rename npc and type a Vec
matt-o-how Jul 30, 2024
2c8a045
Revert "rename multithread_validation to spend_bundle_validation"
matt-o-how Jul 30, 2024
5ff17d2
remove added file
matt-o-how Jul 30, 2024
6ed8437
rename multithread validation
matt-o-how Jul 30, 2024
4b212fd
re-apply updates to spendbundle_validation
matt-o-how Jul 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions crates/chia-consensus/src/multiprocess_validation.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ use crate::gen::opcodes::{
};
use crate::gen::owned_conditions::OwnedSpendBundleConditions;
use crate::gen::validation_error::ErrorCode;
use crate::npc_result::get_name_puzzle_conditions;
use crate::npc_result::get_conditions_from_spendbundle;
use chia_bls::BlsCache;
use chia_protocol::SpendBundle;
use clvmr::{ENABLE_BLS_OPS_OUTSIDE_GUARD, ENABLE_FIXED_DIV};
Expand All @@ -24,8 +24,9 @@ pub fn validate_clvm_and_signature(
cache: &BlsCache,
) -> Result<(OwnedSpendBundleConditions, Duration), ErrorCode> {
let start_time = Instant::now();
let npcresult = get_name_puzzle_conditions(spend_bundle, max_cost, true, height, constants)
.map_err(|e| e.1)?;
let npcresult =
get_conditions_from_spendbundle(spend_bundle, max_cost, true, height, constants)
.map_err(|e| e.1)?;
let iter = npcresult.spends.iter().flat_map(|spend| {
let condition_items_pairs = [
(AGG_SIG_PARENT, &spend.agg_sig_parent),
Expand Down
2 changes: 1 addition & 1 deletion crates/chia-consensus/src/npc_result.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ use clvmr::run_program::run_program;
use clvmr::serde::node_from_bytes;
use std::collections::{HashMap, HashSet};

pub fn get_name_puzzle_conditions(
pub fn get_conditions_from_spendbundle(
spend_bundle: &SpendBundle,
max_cost: u64,
mempool_mode: bool,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this parameter should be removed. You're only dealing with transactions, i.e. mempool mode

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Expand Down
2 changes: 1 addition & 1 deletion wheel/generate_type_stubs.py
Original file line number Diff line number Diff line change
Expand Up @@ -313,7 +313,7 @@ def validate_clvm_and_signature(
cache: Optional[BLSCache],
) -> Tuple[SpendBundleConditions, float]: ...

def get_name_puzzle_conditions(
def get_conditions_from_spendbundle(
spend_bundle: SpendBundle,
max_cost: int,
constants: ConsensusConstants,
Expand Down
2 changes: 1 addition & 1 deletion wheel/python/chia_rs/chia_rs.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ def validate_clvm_and_signature(
cache: Optional[BLSCache],
) -> Tuple[SpendBundleConditions, float]: ...

def get_name_puzzle_conditions(
def get_conditions_from_spendbundle(
spend_bundle: SpendBundle,
max_cost: int,
constants: ConsensusConstants,
Expand Down
19 changes: 10 additions & 9 deletions wheel/src/api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ use chia_consensus::merkle_set::compute_merkle_set_root as compute_merkle_root_i
use chia_consensus::merkle_tree::{validate_merkle_proof, MerkleSet};
use chia_consensus::multiprocess_validation::validate_clvm_and_signature;

use chia_consensus::npc_result::get_name_puzzle_conditions;
use chia_consensus::npc_result::get_conditions_from_spendbundle;
use chia_protocol::{
BlockRecord, Bytes32, ChallengeBlockInfo, ChallengeChainSubSlot, ClassgroupElement, Coin,
CoinSpend, CoinState, CoinStateFilters, CoinStateUpdate, EndOfSubSlotBundle, Foliage,
Expand Down Expand Up @@ -398,19 +398,20 @@ pub fn py_validate_clvm_and_signature(
}

#[pyfunction]
#[pyo3(name = "get_name_puzzle_conditions")]
pub fn py_get_name_puzzle_conditions(
#[pyo3(name = "get_conditions_from_spendbundle")]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to call this from python?
It would also need to be updated in chia_rs.pyi

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. It is required to fix a bug with the fast-forwarding code

pub fn py_get_conditions_from_spendbundle(
spend_bundle: &SpendBundle,
max_cost: u64,
constants: &ConsensusConstants,
mempool_mode: bool,
height: u32,
) -> PyResult<OwnedSpendBundleConditions> {
let osbc = get_name_puzzle_conditions(spend_bundle, max_cost, mempool_mode, height, constants)
.map_err(|e| {
let error_code: u32 = e.1.into();
PyErr::new::<PyTypeError, _>(error_code)
})?;
let osbc =
get_conditions_from_spendbundle(spend_bundle, max_cost, mempool_mode, height, constants)
.map_err(|e| {
let error_code: u32 = e.1.into();
PyErr::new::<PyTypeError, _>(error_code)
})?;
Ok(osbc)
}

Expand Down Expand Up @@ -445,7 +446,7 @@ pub fn chia_rs(_py: Python<'_>, m: &Bound<'_, PyModule>) -> PyResult<()> {

// multithread validattion
m.add_function(wrap_pyfunction!(py_validate_clvm_and_signature, m)?)?;
m.add_function(wrap_pyfunction!(py_get_name_puzzle_conditions, m)?)?;
m.add_function(wrap_pyfunction!(py_get_conditions_from_spendbundle, m)?)?;

// clvm functions
m.add("NO_UNKNOWN_CONDS", NO_UNKNOWN_CONDS)?;
Expand Down