Skip to content

Commit 8558c54

Browse files
madlabmanF4evershuoer86vgorkavenko
authored
CSM performance oracle module (#440)
* add label to FRAME_PREV_REPORT_REF_SLOT * add withdrawal exit epoch * add labels to CONTRACT_ON_PAUSE * linter fix * fix typo * fix negative block on forks * Fix typo in docs * Fix typo in test function name * Fix typo in test function name * feat: attestation and committees * fix: type * fix: return -> yield * fix: pylint in tests * fix: mypy issues * fix: test * chore: change typing for clients * refactor: strict way for streams `_get_without_fallbacks` * chore: remove non-mandatory pylint ignore * make dockerfile buildable * feat: add CSM performance oracle module * contracts create * temp changes * rename typings * fix get_latest_blockstamp * fix test consensus * tests fixes * run black * fix linters * add blockhash * fix tests * fix mypy * fix linter * feat: CSM contracts as a separate module * fix comments * pylint fixed * fix lru cache * test for cache * fix linter * fix linters * remove unused import * fix: off-by-one error on logs fetching * chore: op_id -> no_id * feat: lazy-loading of CSM module * fix: get ABIs for contracts * chore: update docs for env variables * docs: few more corrections to README * chore: fix timeout_decorator type stub * fix: remove lru_cache from CSFeeOracle * feat: collect data for report building * fix: processing * fix: review * fix: ignore * feat: CSM contracts as a separate module * fix: off-by-one error on logs fetching * chore: op_id -> no_id * feat: lazy-loading of CSM module * fix: get ABIs for contracts * chore: update docs for env variables * docs: few more corrections to README * chore: fix timeout_decorator type stub * fix: remove lru_cache from CSFeeOracle * fix: lint * unify get call function * back to HexBytes * feat: add dummy IPFS provider (#436) * feat: `get_module_validators_by_node_operators` (#438) * feat: build CSM oracle report (#437) * fix: calculated l_ref_slot and r_ref_slot (#441) * fix: calculated l_ref_slot and r_ref_slot * fix: calculated l_ref_slot and r_ref_slot * CSM oracle hot fixes (#442) * feat: handle thread exception in processing (#443) * feat: handle thread exception in processing * fix: remove unused things * fix: use `ValueError`, shutdown the executor * fix: linter * chore: add a comment * fix: rise `SystemExit` (#446) * fix: put a stone into a tree (#444) * feat: `epochs_to_process` (#445) * feat: `epochs_to_process` * feat: two sets * feat: advanced validation * fix: use `unprocessed_epochs` * fix: private state properties * revert: `_data` -> ``data` * feat: gw3.io provider and MultiIPFSProvider (#447) * Fix report (#448) * fix: checkpoint range assert (#449) * fix: use reference blockstamp as r_slot for stuck events search (#450) * feat: use attestations endpoint * feat: fetch stuck events for the given frame only (#452) * feat: configure workers count for CSM module * chore: remove DummyIPFSProvider from main * fix: tests import * fix: csm `module` property * fix: logging CID via repr * chore: remove stone when not needed * fix: submit_report_data accepts a tuple * chore: fix mypy * feat: implement dynamic versions * fix: black: two lines * replace wget to culr * fix: call get_contract_version with block hash * Close some TODOs (#456) * fix message * chore: move iterator to folder * chore: add timestamp to logs * feat: add tests to lido contracts * fix: test slot * fix pylint * black fix * update workflow * chore: add v2 iterator * fix tests * fix: review * chore: move iterator to folder * chore: add timestamp to logs * feat: add tests to lido contracts * fix: test slot * fix pylint * black fix * update workflow * fix: review * fix: pylint * feat: absolute performance leeway * fix integration tests * iterator update * fix tests * fix: iterator test * feat: Multi Third Phase * squash sr abi * chore: use TOTAL_BASIS_POINTS const * fix middleware test * fix test * feat: ao-multi-third-phase-partial * fix: review * fix tests * fix: cast to tuple issue * fix log * fix: first validator issue * fix pylint * fix all tests * feat: Multi Third Phase * feat: ao-multi-third-phase-partial * fix: review * fix tests * fix: cast to tuple issue * fix log * tweak ejector * feat: checkpoints refactoring + tests (#463) * feat: review fixes * fix: linters * fix: lru cache * fix: call from v2 old contract * fix: linters * fix: `CSFeeOracle` with no initial epoch * Update src/modules/csm/csm.py Co-authored-by: Don Perignom <[email protected]> * fix: first report l_ref_slot (#466) * fix: error messages for `validate_for_report` (#467) * fix: error messages for `validate_for_report` * fix: use error message explicitly * fix: Add warning log * feat: Update python to 3.12 * fix: State network average calculation * fix: compute operators shares via assigned attestations * refactor: fix linters * refactor: calculate distribution * tests: fix missing assert * refactor: no cache for stuck_operators * fix tests and improve cache * chore: log is_deployed result Co-authored-by: Raman Siamionau <[email protected]> * chore: add CSM_MODULE_ADDRESS to .env.example * chore: add more catch statements for IPFS * chore: add missing variables to README * fix: check contracts via _load_contracts * refactor: CSFeeDistributor calls casts and logs * chore: debug -> info logs on calls in CSM contracts * refactor: add Contract suffix to CSM contracts * Fix: wf * refactor: InconsistentEvents * refactor: move TOTAL_BASIS_POINTS out of perf_leeway * chore: fix sequence assert * chore: update CSModule abi * fix linters * fix: use module address to get keys from keysapi * fix: remove `no_cache` method for committees * fix: merge `get` and `get stream` base methods * feat: check module address * refactor: extract get_events_in_range into utils * fix: rework `get_first_non_missed_slot` * chore: remove events inconsistency check from cs_module * refactor: use get_events_in_range across the codebase * fix: add `Final` for env vars * tests: fix test_calculate_distribution * fix after merge * fix: black * fix: tests * refactor: extract get_operators_with_stucks_in_range * refactor: block -> block_identifier * chore: alias for Shares * fix: get_stuck_operators_ids return * tests: rework events utils tests * fix test names * fix: calculate_distribution shares as a defaultdict * docs: CSM-only variables as CSM only in README * Update src/utils/slot.py * chore: fix logging for get_events_in_range * chore: StakingModuleAddress as ChecksumAddress * chore: update authors * chore: remove redundant cache * refactor: move State.load to __init__ * fix: consensus farFutureEpoch const * chore: add check for l_ref_slot * fix: `minimum checkpoint step is not reached` is not error (#472) * fix: review * fix: test * feat: modify cache * comment problematic integration test * feat: block roots from state, events (#474) * feat: block roots from state, events * fix: check events len * fix: LazyCSM -> CSM * fix: assert * feat: check providers for CSM frame * feat: add DepositContract * chore: fix linters --------- Co-authored-by: madlabman <[email protected]> * fix: linters * fix: `get_attestation_committees` returns list now (#476) * fix: `get_attestation_committees` returns list now * fix: linter * fix: `_get_block_roots` * fix: test * fix: checkpoint slot and tests * fix: use condition from spec * chore: set default concurrency for CSM to "2" * fix: iterator conditions (#477) * fix: iterator conditions * fix: black * fix: use max_available_epoch_to_check * fix: _get_block_roots * fix: black * fix: _select_block_roots * fix: pylint * feat: `batched` * feat: get CSFeeOracle address via CSModule (#478) * feat: check IPFS provider for CSM (#481) * feat: check IPFS provider for CSM * feat: IPFS checker all providers * feat: IPFS checker mark skipped providers * fix: review (#480) * fix: review * fix: linter * fix: tests * fix: no faker for checker --------- Co-authored-by: F4ever <[email protected]> Co-authored-by: shuoer86 <[email protected]> Co-authored-by: vgorkavenko <[email protected]> Co-authored-by: Vladimir Gorkavenko <[email protected]> Co-authored-by: F4ever <[email protected]>
1 parent 0deeede commit 8558c54

File tree

177 files changed

+13423
-3908
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

177 files changed

+13423
-3908
lines changed

.env.example

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,7 @@ CONSENSUS_CLIENT_URI=http://...
22
EXECUTION_CLIENT_URI=http://...
33
KEYS_API_URI=https://...
44
LIDO_LOCATOR_ADDRESS=0x1...
5-
MEMBER_PRIV_KEY=aaa...
5+
CSM_ORACLE_ADDRESS=0x...
6+
CSM_MODULE_ADDRESS=0x...
7+
MEMBER_PRIV_KEY=aaa...
8+

.github/workflows/ci-dev-goerli.yml

Lines changed: 0 additions & 29 deletions
This file was deleted.

.github/workflows/tests.yml

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,10 @@ jobs:
1313
steps:
1414
- uses: actions/checkout@v3
1515

16-
- name: Set up Python 3.11
16+
- name: Set up Python 3.12
1717
uses: actions/setup-python@v4
1818
with:
19-
python-version: "3.11"
19+
python-version: "3.12"
2020

2121
- name: Setup poetry
2222
run: >
@@ -31,18 +31,22 @@ jobs:
3131
poetry install --no-interaction --with=dev
3232
3333
- name: Test with pytest
34-
run: poetry run pytest -m "not integration and not e2e" --cov=src tests
34+
run: poetry run pytest --cov=src tests
35+
env:
36+
EXECUTION_CLIENT_URI: ${{ secrets.EXECUTION_CLIENT_URI }}
37+
CONSENSUS_CLIENT_URI: ${{ secrets.CONSENSUS_CLIENT_URI }}
38+
KEYS_API_URI: ${{ secrets.KEYS_API_URI }}
3539

3640
linters:
3741
runs-on: ubuntu-latest
3842

3943
steps:
4044
- uses: actions/checkout@v3
4145

42-
- name: Set up Python 3.11
46+
- name: Set up Python 3.12
4347
uses: actions/setup-python@v4
4448
with:
45-
python-version: "3.11"
49+
python-version: "3.12"
4650

4751
- name: Setup poetry
4852
run: >

.gitignore

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -126,6 +126,8 @@ venv.bak/
126126
.dmypy.json
127127
dmypy.json
128128

129+
.ruff_cache/
130+
129131
# Pyre type checker
130132
.pyre/
131133

@@ -134,4 +136,7 @@ dmypy.json
134136

135137
# vim
136138
*.swp
137-
.direnv
139+
140+
# Cache
141+
*.pkl
142+
*.buf

Dockerfile

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,9 @@
1-
FROM python:3.11-slim as base
1+
FROM python:3.12.4-slim as base
22

33
RUN apt-get update && apt-get install -y --no-install-recommends -qq \
4-
gcc=4:10.2.1-1 \
5-
libffi-dev=3.3-6 \
6-
g++=4:10.2.1-1 \
7-
curl=7.74.0-1.3+deb11u7 \
4+
libffi-dev=3.4.4-1 \
5+
g++=4:12.2.0-3 \
6+
curl=7.88.1-10+deb12u5 \
87
&& apt-get clean \
98
&& rm -rf /var/lib/apt/lists/*
109

README.md

Lines changed: 68 additions & 45 deletions
Large diffs are not rendered by default.

assets/CSAccounting.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
[{"type":"function","name":"chargeFee","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"amount","type":"uint256","internalType":"uint256"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"chargeRecipient","inputs":[],"outputs":[{"name":"","type":"address","internalType":"address"}],"stateMutability":"view"},{"type":"function","name":"claimRewardsStETH","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"stETHAmount","type":"uint256","internalType":"uint256"},{"name":"rewardAddress","type":"address","internalType":"address"},{"name":"cumulativeFeeShares","type":"uint256","internalType":"uint256"},{"name":"rewardsProof","type":"bytes32[]","internalType":"bytes32[]"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"claimRewardsUnstETH","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"stEthAmount","type":"uint256","internalType":"uint256"},{"name":"rewardAddress","type":"address","internalType":"address"},{"name":"cumulativeFeeShares","type":"uint256","internalType":"uint256"},{"name":"rewardsProof","type":"bytes32[]","internalType":"bytes32[]"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"claimRewardsWstETH","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"wstETHAmount","type":"uint256","internalType":"uint256"},{"name":"rewardAddress","type":"address","internalType":"address"},{"name":"cumulativeFeeShares","type":"uint256","internalType":"uint256"},{"name":"rewardsProof","type":"bytes32[]","internalType":"bytes32[]"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"compensateLockedBondETH","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[],"stateMutability":"payable"},{"type":"function","name":"depositETH","inputs":[{"name":"from","type":"address","internalType":"address"},{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[],"stateMutability":"payable"},{"type":"function","name":"depositStETH","inputs":[{"name":"from","type":"address","internalType":"address"},{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"stETHAmount","type":"uint256","internalType":"uint256"},{"name":"permit","type":"tuple","internalType":"struct ICSAccounting.PermitInput","components":[{"name":"value","type":"uint256","internalType":"uint256"},{"name":"deadline","type":"uint256","internalType":"uint256"},{"name":"v","type":"uint8","internalType":"uint8"},{"name":"r","type":"bytes32","internalType":"bytes32"},{"name":"s","type":"bytes32","internalType":"bytes32"}]}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"depositWstETH","inputs":[{"name":"from","type":"address","internalType":"address"},{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"wstETHAmount","type":"uint256","internalType":"uint256"},{"name":"permit","type":"tuple","internalType":"struct ICSAccounting.PermitInput","components":[{"name":"value","type":"uint256","internalType":"uint256"},{"name":"deadline","type":"uint256","internalType":"uint256"},{"name":"v","type":"uint8","internalType":"uint8"},{"name":"r","type":"bytes32","internalType":"bytes32"},{"name":"s","type":"bytes32","internalType":"bytes32"}]}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"feeDistributor","inputs":[],"outputs":[{"name":"","type":"address","internalType":"contract ICSFeeDistributor"}],"stateMutability":"view"},{"type":"function","name":"getActualLockedBond","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getBond","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getBondAmountByKeysCount","inputs":[{"name":"keys","type":"uint256","internalType":"uint256"},{"name":"curve","type":"tuple","internalType":"struct ICSBondCurve.BondCurve","components":[{"name":"points","type":"uint256[]","internalType":"uint256[]"},{"name":"trend","type":"uint256","internalType":"uint256"}]}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getBondAmountByKeysCount","inputs":[{"name":"keys","type":"uint256","internalType":"uint256"},{"name":"curveId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getBondAmountByKeysCountWstETH","inputs":[{"name":"keysCount","type":"uint256","internalType":"uint256"},{"name":"curveId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getBondAmountByKeysCountWstETH","inputs":[{"name":"keysCount","type":"uint256","internalType":"uint256"},{"name":"curve","type":"tuple","internalType":"struct ICSBondCurve.BondCurve","components":[{"name":"points","type":"uint256[]","internalType":"uint256[]"},{"name":"trend","type":"uint256","internalType":"uint256"}]}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getBondCurve","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"tuple","internalType":"struct ICSBondCurve.BondCurve","components":[{"name":"points","type":"uint256[]","internalType":"uint256[]"},{"name":"trend","type":"uint256","internalType":"uint256"}]}],"stateMutability":"view"},{"type":"function","name":"getBondCurveId","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getBondLockRetentionPeriod","inputs":[],"outputs":[{"name":"retention","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getBondShares","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getCurveInfo","inputs":[{"name":"curveId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"tuple","internalType":"struct ICSBondCurve.BondCurve","components":[{"name":"points","type":"uint256[]","internalType":"uint256[]"},{"name":"trend","type":"uint256","internalType":"uint256"}]}],"stateMutability":"view"},{"type":"function","name":"getKeysCountByBondAmount","inputs":[{"name":"amount","type":"uint256","internalType":"uint256"},{"name":"curve","type":"tuple","internalType":"struct ICSBondCurve.BondCurve","components":[{"name":"points","type":"uint256[]","internalType":"uint256[]"},{"name":"trend","type":"uint256","internalType":"uint256"}]}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getKeysCountByBondAmount","inputs":[{"name":"amount","type":"uint256","internalType":"uint256"},{"name":"curveId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getLockedBondInfo","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"tuple","internalType":"struct ICSBondLock.BondLock","components":[{"name":"amount","type":"uint128","internalType":"uint128"},{"name":"retentionUntil","type":"uint128","internalType":"uint128"}]}],"stateMutability":"view"},{"type":"function","name":"getRequiredBondForNextKeys","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"additionalKeys","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getRequiredBondForNextKeysWstETH","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"additionalKeys","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getUnbondedKeysCount","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getUnbondedKeysCountToEject","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"lockBondETH","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"amount","type":"uint256","internalType":"uint256"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"penalize","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"amount","type":"uint256","internalType":"uint256"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"releaseLockedBondETH","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"amount","type":"uint256","internalType":"uint256"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"resetBondCurve","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"setBondCurve","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"curveId","type":"uint256","internalType":"uint256"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"settleLockedBondETH","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"nonpayable"},{"type":"function","name":"totalBondShares","inputs":[],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"}]

0 commit comments

Comments
 (0)