Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add functions for multithreaded validation to the Rust side. #541

Open
wants to merge 122 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 105 commits
Commits
Show all changes
122 commits
Select commit Hold shift + click to select a range
9628d74
port errors and npc_result to rust
matt-o-how Apr 25, 2024
b1d414e
add function bones for filling in
matt-o-how Apr 25, 2024
8ec9f22
add BlockGenerator and simple_solution_generator
matt-o-how Apr 30, 2024
cbcd4e2
get_flags_for_height_and_constants
matt-o-how Apr 30, 2024
3c8fa30
add get_name_puzzle_conditions
matt-o-how Apr 30, 2024
3a7d6d5
type fixes
matt-o-how May 1, 2024
e02aa6c
more type fixes
matt-o-how May 2, 2024
f6896c9
pkm_pairs
matt-o-how May 6, 2024
2ce2d7b
type fixes
matt-o-how May 6, 2024
3da2738
use blscache
matt-o-how May 15, 2024
844751d
remove references to npcresult
matt-o-how May 16, 2024
8f8eace
minor fixes and improvements
matt-o-how May 20, 2024
68ec039
add syncing toggle
matt-o-how May 20, 2024
424d6e4
makes errs pub
matt-o-how May 22, 2024
07cf8b4
commit before rebase
matt-o-how May 23, 2024
4f6afb8
move to use ConsensusConstants and cargo fixes
matt-o-how May 24, 2024
f3be475
more cargo fixes
matt-o-how May 24, 2024
e24026c
even more cargo fixes
matt-o-how May 24, 2024
e4d518e
continued cargo work
matt-o-how May 24, 2024
46ee479
more compilation error fixes
matt-o-how May 28, 2024
1fe83b9
fix make_aggsig_final_message
matt-o-how May 28, 2024
cecee94
remove unnecessary clones
matt-o-how May 28, 2024
2c12d92
yet more cargo fixes
matt-o-how May 29, 2024
7a9f8ea
handle other opcode cases
matt-o-how May 29, 2024
d18120c
fix simple_solution_generator
matt-o-how May 29, 2024
4949012
fmt
matt-o-how May 29, 2024
088e834
test runs!
matt-o-how May 30, 2024
b91f098
pass test
matt-o-how May 30, 2024
dcbf2e3
fmt
matt-o-how May 30, 2024
6467689
clippy fixes
matt-o-how May 30, 2024
c754324
pass constants as reference
matt-o-how May 30, 2024
0fb2322
remove unnecessary clones
matt-o-how May 30, 2024
a16b487
use ? instead of match
matt-o-how May 30, 2024
6d2a79d
another fmt
matt-o-how May 30, 2024
4a783f0
small arvid suggestions
matt-o-how May 31, 2024
311c860
remove BlockGenerator from get_name_puzzle_conditions
matt-o-how May 31, 2024
d023c2a
reduce clones in loop
matt-o-how May 31, 2024
948ea39
save an allocation
matt-o-how May 31, 2024
8ff50f0
unused import
matt-o-how May 31, 2024
9c80a65
remove block validation function stubs
matt-o-how Jun 3, 2024
c5053fc
remove calls to run_block_generator in get_npc
matt-o-how Jun 3, 2024
c3b5d92
no longer create a generator program
matt-o-how Jun 4, 2024
30003a7
clippy fixes
matt-o-how Jun 4, 2024
42e4b09
use consenusconstants for additional_data instead of hashmap
matt-o-how Jun 4, 2024
4399e8a
switch pkm_pairs to return an iter
matt-o-how Jun 4, 2024
c01bcbe
optimise via inlining pkm_pairs
matt-o-how Jun 5, 2024
24ca094
fmt + clippy
matt-o-how Jun 5, 2024
42b29b0
add unsafes to pkm_iter and add test for it
matt-o-how Jun 6, 2024
fd4a8d8
remove further clones inside iter
matt-o-how Jun 7, 2024
2607c70
add aggsig_me test
matt-o-how Jun 7, 2024
20cb94d
Update crates/chia-consensus/src/multiprocess_validation.rs
matt-o-how Jun 7, 2024
6a54ea0
update get_flags and fix npcresult unwrap
matt-o-how Jun 7, 2024
2a2f354
Use only one aggregate_verify with a (pk, msg) iter
matt-o-how Jun 7, 2024
a0e35df
fmt
matt-o-how Jun 7, 2024
2b90681
remove useless vec cast
matt-o-how Jun 7, 2024
0b5064f
remove syncing, return additions
matt-o-how Jun 10, 2024
d96dab2
fix cache benchmark
matt-o-how Jun 10, 2024
3775939
add aggsig parent puzzle test and remove syncing and fix cache benchmark
matt-o-how Jun 10, 2024
a4ae257
test_validate_aggsig_parent_amount
matt-o-how Jun 10, 2024
d7a2d8d
add test_validate_aggsig_puzzle_amount
matt-o-how Jun 10, 2024
3845a2c
add pyfunction for validate_clvm_and_signature
matt-o-how Jun 11, 2024
af104f6
OwnedSpendBundleConditions never fails now
matt-o-how Jun 17, 2024
b5cf9c2
remove NO_RELATIVE_CONDITIONS_ON_EPHEMERAL flag as it is now uncondit…
matt-o-how Jun 17, 2024
dc53e40
cargo and fmt fixes
matt-o-how Jun 17, 2024
9f69b59
reduce allocations in make_aggsig_final_message
matt-o-how Jun 17, 2024
c0886f7
use hex!
matt-o-how Jun 17, 2024
d453405
add cache.update() and return additions as vec
matt-o-how Jun 18, 2024
e11152b
clippy and fmt
matt-o-how Jun 18, 2024
bea5393
Update crates/chia-consensus/src/multiprocess_validation.rs
matt-o-how Jun 18, 2024
62d1e89
delete BlockGenerator struct
matt-o-how Jun 18, 2024
c6a022d
apply some of arvid's suggestions
matt-o-how Jun 18, 2024
fc1d307
add test to u64_to_bytes
matt-o-how Jun 18, 2024
20c2775
implement more of arvid's comments
matt-o-how Jun 18, 2024
38a08ff
unused import
matt-o-how Jun 18, 2024
33cbfbc
remove unused pyo3 import
matt-o-how Jun 24, 2024
7e84370
add pytest and type stubs
matt-o-how Jun 25, 2024
1654ebf
if no cache passed in, then create one
matt-o-how Jul 8, 2024
d4d0c23
explicitly call conversion
matt-o-how Jul 8, 2024
bfc24ab
cargo fmt
matt-o-how Jul 9, 2024
e7297da
add costing test
matt-o-how Jul 9, 2024
6ed3814
fix DedupFlags not setting properly
matt-o-how Jul 9, 2024
ff30619
return cache additions as bytes
matt-o-how Jul 9, 2024
61697a2
return badaggsig instead of generic badspendbundle
matt-o-how Jul 10, 2024
3280753
create pybinding for get_name_puzzle_conditions
matt-o-how Jul 15, 2024
6230278
clippy
matt-o-how Jul 16, 2024
b90a72c
add type stubs for get_name_puzzle_conditions
matt-o-how Jul 17, 2024
4fb81b5
pass whole GTElement and update with iter
matt-o-how Jul 18, 2024
83d7c5a
Merge branch 'multithread_validation' of https://github.com/Chia-Netw…
matt-o-how Jul 18, 2024
e636637
python items() returns GTElement
matt-o-how Jul 18, 2024
dc46758
use chia_rs values
matt-o-how Jul 19, 2024
e625725
use ARC pointer and don't clone test list
matt-o-how Jul 22, 2024
3359fa9
return python object for GTElement instead of bytes when calling BLSC…
matt-o-how Jul 22, 2024
3861339
black test_blscache.py
matt-o-how Jul 22, 2024
211b63a
stubs fix
matt-o-how Jul 22, 2024
574521b
optimise tests
matt-o-how Jul 23, 2024
451302e
Merge branch 'main' into multithread_validation
matt-o-how Jul 24, 2024
029b120
fix removal of softfork flag
matt-o-how Jul 25, 2024
8768baf
add Additions and ValidationResult types for Clippy's sake
matt-o-how Jul 25, 2024
54318db
Add interior mutability to BLSCache
Rigidity Jul 22, 2024
facd7b5
fix final pytest
matt-o-how Jul 23, 2024
a91aa35
black tests
matt-o-how Jul 23, 2024
3438e6e
Merge pull request #623 from Chia-Network/cache-interior-mutability
matt-o-how Jul 26, 2024
3825fc3
remove commented out code
matt-o-how Jul 26, 2024
abc0366
fmt
matt-o-how Jul 26, 2024
d8192cc
remove .collect() call from validate_clvm_and_signature
matt-o-how Jul 26, 2024
cd48403
remove pre-validate spendbundle
matt-o-how Jul 29, 2024
3b14011
move Arc import to tests mod
matt-o-how Jul 29, 2024
7565924
rename get_name_puzzle_conditions to get_conditions_from_spendbundle
matt-o-how Jul 29, 2024
20f95ce
rename multithread_validation to spend_bundle_validation
matt-o-how Jul 29, 2024
d4fb13f
remove mempool mode bool
matt-o-how Jul 29, 2024
44b5dfc
expose get_flags_for_height_and_constants
matt-o-how Jul 29, 2024
bd307b6
move large spendbundle test to a clsp.hex file
matt-o-how Jul 29, 2024
09e4a8a
add regression test to make_aggsig_final_message
matt-o-how Jul 29, 2024
ec77718
clippy fixes
matt-o-how Jul 29, 2024
fced1e3
rename npc_result to spendbundle_conditions
matt-o-how Jul 30, 2024
47c23d7
remove need for mut in flags
matt-o-how Jul 30, 2024
019b2f1
Revert "rename get_name_puzzle_conditions to get_conditions_from_spen…
matt-o-how Jul 30, 2024
48c706a
re-rename npc and type a Vec
matt-o-how Jul 30, 2024
2c8a045
Revert "rename multithread_validation to spend_bundle_validation"
matt-o-how Jul 30, 2024
5ff17d2
remove added file
matt-o-how Jul 30, 2024
6ed8437
rename multithread validation
matt-o-how Jul 30, 2024
4b212fd
re-apply updates to spendbundle_validation
matt-o-how Jul 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -135,3 +135,4 @@ zstd = "0.13.2"
blocking-threadpool = "1.0.1"
libfuzzer-sys = "0.4"
wasm-bindgen = "0.2.92"
parking_lot = "0.12.3"
1 change: 1 addition & 0 deletions crates/chia-bls/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ thiserror = { workspace = true }
pyo3 = { workspace = true, features = ["multiple-pymethods"], optional = true }
arbitrary = { workspace = true, optional = true }
lru = { workspace = true }
parking_lot = { workspace = true }

[dev-dependencies]
rand = { workspace = true }
Expand Down
27 changes: 11 additions & 16 deletions crates/chia-bls/benches/cache.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,55 +16,50 @@ fn cache_benchmark(c: &mut Criterion) {

let mut agg_sig = Signature::default();
for i in 0..1000 {
let derived = sk.derive_hardened(i as u32);
let derived = sk.derive_hardened(i);
let pk = derived.public_key();
let sig = sign(&derived, msg);
agg_sig.aggregate(&sig);
pks.push(pk);
}

let mut bls_cache = BlsCache::default();
let bls_cache = BlsCache::default();

c.bench_function("bls_cache.aggregate_verify, 0% cache hits", |b| {
let mut cache = bls_cache.clone();
b.iter(|| {
assert!(cache.aggregate_verify(&pks, [&msg].iter().cycle(), &agg_sig));
assert!(bls_cache.aggregate_verify(pks.iter().zip([&msg].iter().cycle()), &agg_sig));
});
});

// populate 10% of keys
bls_cache.aggregate_verify(&pks[0..100], [&msg].iter().cycle(), &agg_sig);
bls_cache.aggregate_verify(pks[0..100].iter().zip([&msg].iter().cycle()), &agg_sig);
c.bench_function("bls_cache.aggregate_verify, 10% cache hits", |b| {
let mut cache = bls_cache.clone();
b.iter(|| {
assert!(cache.aggregate_verify(&pks, [&msg].iter().cycle(), &agg_sig));
assert!(bls_cache.aggregate_verify(pks.iter().zip([&msg].iter().cycle()), &agg_sig));
});
});

// populate another 10% of keys
bls_cache.aggregate_verify(&pks[100..200], [&msg].iter().cycle(), &agg_sig);
bls_cache.aggregate_verify(pks[100..200].iter().zip([&msg].iter().cycle()), &agg_sig);
c.bench_function("bls_cache.aggregate_verify, 20% cache hits", |b| {
let mut cache = bls_cache.clone();
b.iter(|| {
assert!(cache.aggregate_verify(&pks, [&msg].iter().cycle(), &agg_sig));
assert!(bls_cache.aggregate_verify(pks.iter().zip([&msg].iter().cycle()), &agg_sig));
});
});

// populate another 30% of keys
bls_cache.aggregate_verify(&pks[200..500], [&msg].iter().cycle(), &agg_sig);
bls_cache.aggregate_verify(pks[200..500].iter().zip([&msg].iter().cycle()), &agg_sig);
c.bench_function("bls_cache.aggregate_verify, 50% cache hits", |b| {
let mut cache = bls_cache.clone();
b.iter(|| {
assert!(cache.aggregate_verify(&pks, [&msg].iter().cycle(), &agg_sig));
assert!(bls_cache.aggregate_verify(pks.iter().zip([&msg].iter().cycle()), &agg_sig));
});
});

// populate all other keys
bls_cache.aggregate_verify(&pks[500..1000], [&msg].iter().cycle(), &agg_sig);
bls_cache.aggregate_verify(pks[500..1000].iter().zip([&msg].iter().cycle()), &agg_sig);
c.bench_function("bls_cache.aggregate_verify, 100% cache hits", |b| {
let mut cache = bls_cache.clone();
b.iter(|| {
assert!(cache.aggregate_verify(&pks, [&msg].iter().cycle(), &agg_sig));
assert!(bls_cache.aggregate_verify(pks.iter().zip([&msg].iter().cycle()), &agg_sig));
});
});

Expand Down
86 changes: 39 additions & 47 deletions crates/chia-bls/src/bls_cache.rs
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
use std::borrow::Borrow;
use std::num::NonZeroUsize;

use lru::LruCache;
use sha2::{Digest, Sha256};

use crate::{aggregate_verify_gt, hash_to_g2};
use crate::{GTElement, PublicKey, Signature};
use lru::LruCache;
use parking_lot::Mutex;
use sha2::{Digest, Sha256};
use std::borrow::Borrow;
use std::num::NonZeroUsize;

/// This is a cache of pairings of public keys and their corresponding message.
/// It accelerates aggregate verification when some public keys have already
Expand All @@ -17,10 +16,10 @@ use crate::{GTElement, PublicKey, Signature};
/// aggregate_verify() primitive is faster. When long-syncing, that's
/// preferable.
#[cfg_attr(feature = "py-bindings", pyo3::pyclass(name = "BLSCache"))]
#[derive(Debug, Clone)]
#[derive(Debug)]
pub struct BlsCache {
// sha256(pubkey + message) -> GTElement
cache: LruCache<[u8; 32], GTElement>,
cache: Mutex<LruCache<[u8; 32], GTElement>>,
}

impl Default for BlsCache {
Expand All @@ -32,47 +31,46 @@ impl Default for BlsCache {
impl BlsCache {
pub fn new(cache_size: NonZeroUsize) -> Self {
Self {
cache: LruCache::new(cache_size),
cache: Mutex::new(LruCache::new(cache_size)),
}
}

pub fn len(&self) -> usize {
self.cache.len()
self.cache.lock().len()
}

pub fn is_empty(&self) -> bool {
self.cache.is_empty()
self.cache.lock().is_empty()
}

pub fn aggregate_verify(
&mut self,
pks: impl IntoIterator<Item = impl Borrow<PublicKey>>,
msgs: impl IntoIterator<Item = impl AsRef<[u8]>>,
pub fn aggregate_verify<Pk: Borrow<PublicKey>, Msg: AsRef<[u8]>>(
&self,
pks_msgs: impl IntoIterator<Item = (Pk, Msg)>,
sig: &Signature,
) -> bool {
let iter = pks.into_iter().zip(msgs).map(|(pk, msg)| -> GTElement {
let iter = pks_msgs.into_iter().map(|(pk, msg)| -> GTElement {
// Hash pubkey + message
let mut hasher = Sha256::new();
hasher.update(pk.borrow().to_bytes());
hasher.update(msg.as_ref());
let hash: [u8; 32] = hasher.finalize().into();

// If the pairing is in the cache, we don't need to recalculate it.
if let Some(pairing) = self.cache.get(&hash).cloned() {
if let Some(pairing) = self.cache.lock().get(&hash).cloned() {
return pairing;
}

// Otherwise, we need to calculate the pairing and add it to the cache.
let mut aug_msg = pk.borrow().to_bytes().to_vec();
aug_msg.extend_from_slice(msg.as_ref());
aug_msg.extend(msg.as_ref());
let aug_hash = hash_to_g2(&aug_msg);

let mut hasher = Sha256::new();
hasher.update(&aug_msg);
let hash: [u8; 32] = hasher.finalize().into();

let pairing = aug_hash.pair(pk.borrow());
self.cache.put(hash, pairing.clone());
self.cache.lock().put(hash, pairing.clone());
pairing
});

Expand All @@ -83,7 +81,6 @@ impl BlsCache {
#[cfg(feature = "py-bindings")]
use pyo3::{
exceptions::PyValueError,
pybacked::PyBackedBytes,
types::{PyAnyMethods, PyList},
Bound, PyObject, PyResult,
};
Expand Down Expand Up @@ -122,9 +119,9 @@ impl BlsCache {
let msgs = msgs
.iter()?
.map(|item| item?.extract())
.collect::<PyResult<Vec<PyBackedBytes>>>()?;
.collect::<PyResult<Vec<Vec<u8>>>>()?;

Ok(self.aggregate_verify(pks, msgs, sig))
Ok(self.aggregate_verify(pks.into_iter().zip(msgs), sig))
}

#[pyo3(name = "len")]
Expand All @@ -137,27 +134,22 @@ impl BlsCache {
use pyo3::prelude::*;
use pyo3::types::PyBytes;
let ret = PyList::empty_bound(py);
for (key, value) in &self.cache {
ret.append((
PyBytes::new_bound(py, key),
PyBytes::new_bound(py, &value.to_bytes()),
))?;
let cache = self.cache.lock();
for (key, value) in cache.iter() {
ret.append((PyBytes::new_bound(py, key), value.clone().into_py(py)))?;
}
Ok(ret.into())
}

#[pyo3(name = "update")]
pub fn py_update(&mut self, other: &Bound<'_, PyList>) -> PyResult<()> {
let mut cache = self.cache.lock();
for item in other.borrow().iter()? {
let (key, value): (Vec<u8>, Vec<u8>) = item?.extract()?;
self.cache.put(
let (key, value): (Vec<u8>, GTElement) = item?.extract()?;
cache.put(
key.try_into()
.map_err(|_| PyValueError::new_err("invalid key"))?,
GTElement::from_bytes(
(&value[..])
.try_into()
.map_err(|_| PyValueError::new_err("invalid GTElement"))?,
),
value,
);
}
Ok(())
Expand All @@ -173,7 +165,7 @@ pub mod tests {

#[test]
fn test_aggregate_verify() {
let mut bls_cache = BlsCache::default();
let bls_cache = BlsCache::default();

let sk = SecretKey::from_seed(&[0; 32]);
let pk = sk.public_key();
Expand All @@ -187,17 +179,17 @@ pub mod tests {
assert!(bls_cache.is_empty());

// Verify the signature and add to the cache.
assert!(bls_cache.aggregate_verify(pk_list, msg_list, &sig));
assert!(bls_cache.aggregate_verify(pk_list.into_iter().zip(msg_list), &sig));
assert_eq!(bls_cache.len(), 1);

// Now that it's cached, it shouldn't cache it again.
assert!(bls_cache.aggregate_verify(pk_list, msg_list, &sig));
assert!(bls_cache.aggregate_verify(pk_list.into_iter().zip(msg_list), &sig));
assert_eq!(bls_cache.len(), 1);
}

#[test]
fn test_cache() {
let mut bls_cache = BlsCache::default();
let bls_cache = BlsCache::default();

let sk1 = SecretKey::from_seed(&[0; 32]);
let pk1 = sk1.public_key();
Expand All @@ -211,7 +203,7 @@ pub mod tests {
assert!(bls_cache.is_empty());

// Add the first signature to cache.
assert!(bls_cache.aggregate_verify(&pk_list, &msg_list, &agg_sig));
assert!(bls_cache.aggregate_verify(pk_list.iter().zip(msg_list.iter()), &agg_sig));
assert_eq!(bls_cache.len(), 1);

// Try with the first key message pair in the cache but not the second.
Expand All @@ -223,7 +215,7 @@ pub mod tests {
pk_list.push(pk2);
msg_list.push(msg2);

assert!(bls_cache.aggregate_verify(&pk_list, &msg_list, &agg_sig));
assert!(bls_cache.aggregate_verify(pk_list.iter().zip(msg_list.iter()), &agg_sig));
assert_eq!(bls_cache.len(), 2);

// Try reusing a public key.
Expand All @@ -234,14 +226,14 @@ pub mod tests {
msg_list.push(msg3);

// Verify this signature and add to the cache as well (since it's still a different aggregate).
assert!(bls_cache.aggregate_verify(pk_list, msg_list, &agg_sig));
assert!(bls_cache.aggregate_verify(pk_list.iter().zip(msg_list), &agg_sig));
assert_eq!(bls_cache.len(), 3);
}

#[test]
fn test_cache_limit() {
// The cache is limited to only 3 items.
let mut bls_cache = BlsCache::new(NonZeroUsize::new(3).unwrap());
let bls_cache = BlsCache::new(NonZeroUsize::new(3).unwrap());

// Before we cache anything, it should be empty.
assert!(bls_cache.is_empty());
Expand All @@ -257,11 +249,11 @@ pub mod tests {
let msg_list = [msg];

// Add to cache by validating them one at a time.
assert!(bls_cache.aggregate_verify(pk_list.iter(), msg_list.iter(), &sig));
assert!(bls_cache.aggregate_verify(pk_list.into_iter().zip(msg_list), &sig));
}

// The cache should be full now.
assert_eq!(bls_cache.cache.len(), 3);
assert_eq!(bls_cache.len(), 3);

// Recreate first key.
let sk = SecretKey::from_seed(&[1; 32]);
Expand All @@ -275,16 +267,16 @@ pub mod tests {
let hash: [u8; 32] = hasher.finalize().into();

// The first key should have been removed, since it's the oldest that's been accessed.
assert!(!bls_cache.cache.contains(&hash));
assert!(!bls_cache.cache.lock().contains(&hash));
}

#[test]
fn test_empty_sig() {
let mut bls_cache = BlsCache::default();
let bls_cache = BlsCache::default();

let pks: [&PublicKey; 0] = [];
let msgs: [&[u8]; 0] = [];

assert!(bls_cache.aggregate_verify(pks, msgs, &Signature::default()));
assert!(bls_cache.aggregate_verify(pks.into_iter().zip(msgs), &Signature::default()));
}
}
Loading
Loading