-
Notifications
You must be signed in to change notification settings - Fork 623
feat(permissionless batches): batch production toolkit and operator recovery #1555
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 147 commits
Commits
Show all changes
153 commits
Select commit
Hold shift + click to select a range
dca69ce
implement first steps for minimal recovery to permissionlessly produc…
jonastheis 2e09118
Merge remote-tracking branch 'origin/develop' into jt/permissionless-…
jonastheis 496314f
add config for recovery mode
jonastheis c12d380
structure code and implement restoreMinimalPreviousState and fetchL2B…
jonastheis c329959
produce chunks from specified L2 blocks and batch from chunks
jonastheis 71f240b
start implementation of restoring full previous state for relayer
jonastheis acc7083
implement processFinalizedBatch
jonastheis 59ea991
handle batches
jonastheis 2df07a9
introduce ForceL1MessageCount to config to be able to set a custom L1…
jonastheis 0f5ebf3
add Dockerfile for relayer in permissionless batches mode
jonastheis f96af8e
add docker compose file to spin up all necessary services together
jonastheis 16a471d
move docker file to build/dockerfiles
jonastheis d25094b
add coordinator-cron and proving-service-bundle
jonastheis 603feed
add bundle creation
jonastheis 5ddd6d6
add permissionless-batches/conf/ to dockerignore files
jonastheis 5ff6fd0
refactor cmd/permissionless_batches/app
jonastheis 0108873
implement RecoveryNeeded functionality to allow re-running without ov…
jonastheis a6f914a
refactor cmd/rollup_relayer/app and split into FullRecovery struct
jonastheis 0123502
clean up
jonastheis 596d9fe
introduce profiles to docker-compose.yml
jonastheis d85bdf5
initial instructions in README.md
jonastheis a9eac08
address review comments
jonastheis 606162e
add dummy configuration and documentation for permissionless batch pr…
jonastheis 30c0201
ignore /conf folder from git
jonastheis ec9d862
add documentation for operator recovery
jonastheis 6ef4775
address review comments
jonastheis 41606fe
support first version of CodecV6 in relayer and add functionality to …
jonastheis 0c0c417
adjust Sender to support multiple blobs when sending a transaction
jonastheis 78c9963
implement batch submission of multiple batches per transaction and co…
jonastheis 940fde0
implement missing part of commit logic for CodecV6
jonastheis ca8d930
feat(rollup-relayer): add Euclid support
omerfirmak 421afe9
deprecate halo2 provers, start openvm prover
omerfirmak b460d4a
Merge remote-tracking branch 'origin/omerfirmak/mpt' into feat/use-co…
jonastheis 182f8e3
update go.mod to latest da-codec version V7
jonastheis 783b965
update go.mod to latest da-codec version V7
jonastheis 5a479c3
address review comments
jonastheis 310abdd
add InitialL1MessageQueueHash and LastL1MessageQueueHash to chunk and…
jonastheis 2efbbd7
add support for bundles with CodecV7
jonastheis e713424
update contextIDFromBatches prefix to v7
jonastheis f4e17bc
fix ci
jonastheis 99c0a9f
update to latest da-codec
jonastheis 8db5339
fix issues with chunks when handling CodecV7
jonastheis 69a80d4
add testCommitBatchAndFinalizeBundleCodecV7 for relayer
jonastheis 5f22950
feat(prover): euclid prover first draft
omerfirmak 9412c7f
feat(coordinator): abstract proof types behind an interface
omerfirmak c3a3bad
feat(rollup-relayer): add Euclid support
omerfirmak 2de45f0
feat(coordinator): add euclid proof types
omerfirmak f01af24
fix: potentially flaky testResubmitZeroGasPriceTransaction
omerfirmak 2bd0655
feat: Add Euclid verifier
omerfirmak 848d3a6
fix: re-enable supported forks check
omerfirmak e5ad9c6
feat: allow stopping fake finalization at fork boundary
omerfirmak b424cef
feat: add CPU prover dockerfile
omerfirmak ac17696
feat: update contracts to euclid version
omerfirmak d503d4a
fix: base64 encode VKs
omerfirmak bf08436
fix: prover using wrong resources for batch/bundle circuits
omerfirmak 83c73f8
fix: coordinator ignore euclid transition chunk
omerfirmak d3acd6b
adjust to recent CodecV7 changes. remove initialL1MessageQueueIndex a…
jonastheis f13863e
Merge remote-tracking branch 'origin/omerfirmak/euclid-prover' into f…
jonastheis 8b57dd6
fixes after merge
jonastheis 37924b0
feat: bump zkvm-prover to rc4
omerfirmak fbc14ac
address review comments
jonastheis 867fda6
fix: add batch tasks details for v6 codec
omerfirmak 2e9827a
fix: properly propagate verifier errors
omerfirmak 01fa3b3
feat: use verifier-only types in coordinator
omerfirmak 50ebf17
fix: generate evm proofs for bundles
omerfirmak e1a0bab
add new contract ABI and adjust submission to it
jonastheis bb9d404
fix: force byte arrays to be marshaled as JSON arrays
omerfirmak 0125dd6
add new contract ABI
jonastheis 121ce09
update config and adjust to new contract ABI
jonastheis ed394a6
make sure that all batches committed in the same tx are part of the s…
jonastheis a55de1f
feat: set code tx support (#1600)
colinlyguo 89ede0d
build rollup images with --platform=linux/amd64
jonastheis 782e019
Merge branch 'feat/use-codec-v6' of github.com:scroll-tech/scroll int…
jonastheis 081d289
update da-codec dependency
jonastheis 47a6c23
fix bug where chunk and batch blocks mismatch
jonastheis eb5758b
feat: bump to zkvm-prover rc5
roynalnaruto 7353f30
update l2geth version in go.mod
jonastheis 673777f
use go 1.22 in Dockerfile builder
jonastheis 554a233
refactor: move euclid prover to new subdir
omerfirmak e27ab5a
Merge remote-tracking branch 'origin/develop' into omerfirmak/euclid-…
omerfirmak 26a49cb
Merge remote-tracking branch 'origin/omerfirmak/euclid-prover' into f…
jonastheis 8ea4315
update go.work.sum
jonastheis 55b32e1
add debug log message
jonastheis d6674e8
add configuration parameter maxChunksPerBatch for batch proposer
jonastheis a1c4562
add configuration parameter maxChunksPerBatch for batch proposer
jonastheis c8b614f
unit tests fix
f91c999
fix batch proposer panic
Thegaram 82dd5e0
feat(bridge-history): support codecv7
4333d51
Revert "feat(bridge-history): support codecv7"
b30f4d0
chore(zkvm-circuit): upgrade to 0.1.0-rc.6 (#1610)
colinlyguo 03c63a6
update batch/chunk details for phase2
omerfirmak 9f4c9ee
fix: prover version (#1611)
yiweichi 101cc46
update dependencies
a868bc1
simplify version checks
8b08a57
Revert "simplify version checks"
f1ea4b3
process task with CodecV7, add check for BlobDataProof length and add…
jonastheis b7e7d1a
Merge remote-tracking branch 'origin/omerfirmak/euclid-prover' into f…
jonastheis 94bee19
feat(bridge-history): support codecv7 (#1609)
colinlyguo f27ddb7
remove debug line
jonastheis da96331
tweak comments and some renamings
d79aaef
fix CI
jonastheis ab7038c
Merge branch 'feat/use-codec-v6' of github.com:scroll-tech/scroll int…
jonastheis 47219f2
run goimports
jonastheis 95adcc3
fix tests
jonastheis 224546e
add new ABI and add more tests
jonastheis 0799dd4
fix linter
jonastheis 7a70e37
fix test
jonastheis 57d50b7
Merge branch 'develop' into omerfirmak/euclid-prover
omerfirmak 20dffe4
chore: auto version bump [bot]
omerfirmak 867307d
Merge remote-tracking branch 'origin/omerfirmak/euclid-prover' into f…
jonastheis c9f6e8c
Merge remote-tracking branch 'origin/develop' into feat/use-codec-v6
jonastheis 9b462e4
go mod tidy
jonastheis d57e6b0
chore: auto version bump [bot]
jonastheis 4d677b3
address review comments
jonastheis 90d1563
Merge remote-tracking branch 'origin/develop' into feat/use-codec-v6
jonastheis d987931
go mod tidy
jonastheis 38b3239
chore: auto version bump [bot]
jonastheis e99a851
Merge branch 'feat/use-codec-v6' of github.com:scroll-tech/scroll int…
jonastheis bb6ee2c
Merge branch 'develop' into feat/use-codec-v6
jonastheis 2d84478
Merge remote-tracking branch 'origin/feat/use-codec-v6' into jt/permi…
jonastheis a8511d5
adjust minimal recovery to changes from CodecV7
jonastheis a16b906
add submission of permissionless batch
jonastheis 942f883
Merge remote-tracking branch 'origin/develop' into jt/permissionless-…
jonastheis 495921c
Merge remote-tracking branch 'origin/develop' into jt/permissionless-…
jonastheis 6414767
update da-codec to latest version
jonastheis 596fabf
update l2geth to latest version
jonastheis 81df41c
add better error handling and wait for transaction confirmation
jonastheis c2152b1
fix issue with replaced l2geth dependency
jonastheis 8009bd1
Merge branch 'develop' into jt/permissionless-batches-recovery
yiweichi 9d54bc2
Merge branch 'develop' into jt/permissionless-batches-recovery
yiweichi ac3305c
Merge branch 'develop' into jt/permissionless-batches-recovery
yiweichi 9d09283
morty/permissionless batches recovery (#1639)
yiweichi d3d5f1f
Merge branch 'develop' into jt/permissionless-batches-recovery
jonastheis 22a6a6e
Merge remote-tracking branch 'origin/develop' into jt/permissionless-…
jonastheis 867c101
implement decoding of codecv7 batches and insertion of chunks, batche…
jonastheis 7531741
goimports
jonastheis ed726d1
go mod tidy
jonastheis 3adafbf
Merge remote-tracking branch 'origin/develop' into jt/permissionless-…
jonastheis aa7df01
go mod tidy
jonastheis 3dde539
Update permissionless-batches/README.md
jonastheis 086542e
Apply suggestions from code review
jonastheis 9569dca
Update permissionless-batches/conf/coordinator/config.json
jonastheis f0af68a
Update rollup/internal/orm/chunk.go
jonastheis b221d42
chore: auto version bump [bot]
jonastheis f102d18
fix: address comments
yiweichi 4f88402
address review comments
jonastheis f663968
Merge branch 'jt/permissionless-batches-recovery' of github.com:scrol…
jonastheis b1ecaf0
fix linter errors
jonastheis 6f36edc
address review comments
jonastheis ee23417
Merge remote-tracking branch 'origin/develop' into jt/permissionless-…
jonastheis 5e5bda9
chore: auto version bump [bot]
jonastheis 1152fb3
update readme with correct link to versions
jonastheis 4e18153
Merge branch 'jt/permissionless-batches-recovery' of github.com:scrol…
jonastheis File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,3 +4,5 @@ docs/ | |
| l2geth/ | ||
| rpc-gateway/ | ||
| *target/* | ||
|
|
||
| permissionless-batches/conf/ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,3 +4,5 @@ docs/ | |
| l2geth/ | ||
| rpc-gateway/ | ||
| *target/* | ||
|
|
||
| permissionless-batches/conf/ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,3 +4,5 @@ docs/ | |
| l2geth/ | ||
| rpc-gateway/ | ||
| *target/* | ||
|
|
||
| permissionless-batches/conf/ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,8 @@ | ||
| assets/ | ||
| contracts/ | ||
| docs/ | ||
| l2geth/ | ||
| rpc-gateway/ | ||
| *target/* | ||
| *target/* | ||
|
|
||
| permissionless-batches/conf/ |
30 changes: 30 additions & 0 deletions
30
build/dockerfiles/recovery_permissionless_batches.Dockerfile
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| # Download Go dependencies | ||
| FROM scrolltech/go-rust-builder:go-1.21-rust-nightly-2023-12-03 as base | ||
|
|
||
| WORKDIR /src | ||
| COPY go.work* ./ | ||
| COPY ./rollup/go.* ./rollup/ | ||
| COPY ./common/go.* ./common/ | ||
| COPY ./coordinator/go.* ./coordinator/ | ||
| COPY ./database/go.* ./database/ | ||
| COPY ./tests/integration-test/go.* ./tests/integration-test/ | ||
| COPY ./bridge-history-api/go.* ./bridge-history-api/ | ||
| RUN go mod download -x | ||
|
|
||
| # Build rollup_relayer | ||
| FROM base as builder | ||
|
|
||
| RUN --mount=target=. \ | ||
| --mount=type=cache,target=/root/.cache/go-build \ | ||
| cd /src/rollup/cmd/permissionless_batches/ && CGO_LDFLAGS="-ldl" go build -v -p 4 -o /bin/rollup_relayer | ||
|
|
||
| # Pull rollup_relayer into a second stage deploy ubuntu container | ||
| FROM ubuntu:20.04 | ||
|
|
||
| RUN apt update && apt install vim netcat-openbsd net-tools curl ca-certificates -y | ||
|
|
||
| ENV CGO_LDFLAGS="-ldl" | ||
|
|
||
| COPY --from=builder /bin/rollup_relayer /bin/ | ||
| WORKDIR /app | ||
| ENTRYPOINT ["rollup_relayer"] |
8 changes: 8 additions & 0 deletions
8
build/dockerfiles/recovery_permissionless_batches.Dockerfile.dockerignore
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| assets/ | ||
| contracts/ | ||
| docs/ | ||
| l2geth/ | ||
| rpc-gateway/ | ||
| *target/* | ||
|
|
||
| permissionless-batches/conf/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,8 @@ | ||
| assets/ | ||
| contracts/ | ||
| docs/ | ||
| l2geth/ | ||
| rpc-gateway/ | ||
| *target/* | ||
| *target/* | ||
|
|
||
| permissionless-batches/conf/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| .PHONY: batch_production_submission launch_prover psql check_proving_status | ||
|
|
||
| export SCROLL_ZKVM_VERSION=0.4.2 | ||
| PG_URL=postgres://postgres@localhost:5432/scroll | ||
|
|
||
| batch_production_submission: | ||
| docker compose --profile batch-production-submission up | ||
|
|
||
| launch_prover: | ||
| docker compose up -d | ||
|
|
||
| psql: | ||
| psql 'postgres://postgres@localhost:5432/scroll' | ||
|
|
||
| check_proving_status: | ||
| @echo "Checking proving status..." | ||
| @result=$$(psql "${PG_URL}" -t -c "SELECT proving_status = 4 AS is_status_success FROM batch ORDER BY index LIMIT 1;" | tr -d '[:space:]'); \ | ||
| if [ "$$result" = "t" ]; then \ | ||
| echo "✅ Prove succeeded! You're ready to submit permissionless batch and proof!"; \ | ||
| else \ | ||
| echo "Proof is not ready..."; \ | ||
| fi | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.