Mainnet 24
This update includes the following changes.
🔒 Network security improvements
Earlier this year, the Flow Foundation published the roadmap to improve Flow network security (see: roadmap). This upgrade includes a major milestone towards attack-resilient message propagation for autonomous node operations. The first version of the application layer spamming protection has now been implemented and applies to all node types. This protection builds on the networking layer protection added in the last upgrade and provides additional protection against a wide range of attacks against a node at the application layer.
Additionally, it also includes further hardening of the networking layer to make it even more resilient to attacks.
Finally, reporting around malicious nodes has been improved as the consensus follower, which runs within all node types, will now detect and report any malicious sender.
These security updates mark a major step towards improving protocol autonomy and ensuring the network is resilient to attacks.
👨💻 Improvements for developers
Simplifying dApp architecture by making it easy to consume transaction events
This upgrade enables event streaming on the REST-based Access API served by the Access nodes. Event streaming makes dApp design simpler by allowing it to consume transaction events using a push-based model instead of a pull-based model. Event streaming is already available using gRPC.
Another enhancement to the Access API in this upgrade is the ability to optionally request the transaction events to be returned in the Cadence Compact Format (CCF) encoding instead of JSON. CCF is a far more compact format and provides better compression of the network payload.
Secure and unlimited script execution
The Access nodes will now also support script execution. This will allow dApps to use a private access node for secure script execution without being subjected to any rate limits. This will also relieve the execution nodes from executing scripts, allowing more CPU on the execution nodes to be allocated towards transaction execution instead. This feature is an alpha release as we continue to test it.
Cadence upgrade
The upgrade will update the supported Cadence version to v0.42.3.
Tamper-proof, verifiable, unpredictable and efficient on-chain random generator
Finally, the update includes partial implementation of FLIP 120: Update unsafeRandom function and FLIP 123: On-chain Random beacon history for commit-reveal schemes. The first FLIP adds a new Cadence function revertibleRandom
that uses the Flow native random beacon to provide secure random numbers for both transactions and scripts. The Flow random beacon provides non-biased, unpredictable and verifiable random numbers, that can be used in Cadence contracts by trusted parties, thanks to the latest FVM updates. The second FLIP implements a solution for reverting random results by untrusted parties. It provides commit-reveal scheme tools, based on the Flow random beacon history. The new revertibleRandom
as well as the new beacon history contract are implemented in this release.
📳 Recovery from the epoch fallback mode
On October 13th, the network went into an epoch fallback mode when it failed to complete an epoch transition. See incident. Consequently, epoch transitions have been suspended, and certain node operations, such as staking and unstaking have also been impacted.
This network upgrade will help the network recover from the epoch fallback mode and resume normal epoch transitions.
💥 There are no breaking changes in this release
Application layer spam protection
- Implements disallowlisting logic by @yhassanzadeh13 in #4441
- Adjust Node Penalty Decay Speed on Repeated Disallow-listing by @gomisha in #4485
- Synchronization Engine
SyncRequest
spam detection (Permissionless-related engine level spam detection) by @gomisha in #4590 - Synchronization Engine
RangeRequest
spam detection (Permissionless-related engine level spam detection) by @gomisha in #4665 - Synchronization Engine
BatchRequest
spam detection (Permissionless-related engine level spam detection) by @gomisha in #4704 - Synchronization Engine spam detection flag support,
config/README.md
updates by @gomisha in #4842 - Khalil/1888 network config improvements by @kc1116 in #4340
- 1891 Gossipsub RPC validation inspector false positive E2E testing by @kc1116 in #4371
- Khalil/1895 Add config package unit test by @kc1116 in #4523
- Khalil/6743 Slashing Violations Consumer ALSP misbehavior report integration by @kc1116 in #4512
- Gossipsub Tracer RPCSent tracker by @kc1116 in #4543
- 1898 slashing violations consumer alsp integration test by @kc1116 in #4549
- Verification Node documentation by @yhassanzadeh13 in #4528
- Khalil/1899 async rpc sent tracker by @kc1116 in #4553
- Khalil/6819 simplify rpc inspector validation by @kc1116 in #4642
Access node
- Do not return not found for tx result when collection not indexed by @peterargue in #4454
- Fix CCF decoding in GetTransactionResultsByBlockID by @peterargue in #4532
- Refactor converters into separate files by @peterargue in #4522
- Add grpc streaming metrics by @peterargue in #4540
- Update REST metrics to use route name for all types by @peterargue in #4571
- Refactor Access RPC engines to support a single gRPC port #4217 by @UlyanaAndrukhiv in #4411
- Connection pool evictions cause connection failures by @Guitarheroua in #4534
- Add circuit breaker for unavailable Collection/Execution nodes by @Guitarheroua in #4496
- Fix slice iteration bug in TrieUpdate protobuf conversion by @peterargue in #4593
- Update default grpc max message size to 1GiB by @peterargue in #4609
- Add endpoint to REST API for retrieving an account key by @m-Peter in #4583
- Make REST server timeouts configurable by @peterargue in #4612
- Script execution engine interface changes by @sideninja in #4668
- Add caching tx results by @nozim in #4598
- Define register index store interface by @sideninja in #4656
- Refactor rpc backend to return an error instead of crashing by @peterargue in #4688
- Add spork and node root block heights to GetNodeVersionInfo by @peterargue in #4690
- Enable Event streaming on REST API by @UlyanaAndrukhiv in #4547
- Pebble height tracking implementation by @koko1123 in #4714
- Upgrade lru cache v2 by @nozim in #4700
- Cleanup access connection management by @peterargue in #4730
- Fix panic condition in client connection cache by @peterargue in #4731
- Index Execution Data by @sideninja in #4653
- Pebble checkpoint ingestion by @koko1123 in #4727
- Index tx results and events by @peterargue in #4772
- Bootstrap register db and start indexer by @peterargue in #4780
- Adjust logging in checkpoint loading by @peterargue in #4797
- Update state stream API to return json-cdc encoded events - backport master by @peterargue in #4803
- Add metrics to execution state indexer by @peterargue in #4801
- Upgrade to grpc v1.58.3 - master by @peterargue in #4810
- Enable grpc compression by @UlyanaAndrukhiv in #4804
- Refactor checkpoint reader with WithFile by @zhangchiqing in #4815
- update execution data protobuf to new namespace by @peterargue in #4827
- Validate checkpoint root hash by @zhangchiqing in #4825
- Add script execution to Access API by @peterargue in #4791
- Validate checkpoint's root hash by @peterargue in #4830
- Remove required Chain flag from execution-state-extract by @janezpodhostnik in #4770
- Add RandomBeaconHistory contract to system transaction by @janezpodhostnik in #4582
- Fix race condition in jobqueue by @peterargue in #4840
- 4215 - Add heartbeat responses to event streaming API by @AndriiDiachuk in #4812
Admin server
- increase range limit by @zhangchiqing in #4476
- add read-range-blocks admin-cmd by @zhangchiqing in #4451
BFT
- Update
math/rand
usage by @tarakby in #4052 - Reporting sender of invalid proposal by @durkmurder in #4364
- Reporting compliance engine protocol violations by @durkmurder in #4350
- Reducing load of BFT tests and improving test packages by @yhassanzadeh13 in #4652
- Decouples BFT Test CI job into smaller CI jobs by @yhassanzadeh13 in #4651
Benchnet
- Remove deprecated --fast-kg from bootstrap by @peterargue in #4619
- Small improvements to the loader to support exec data testing by @peterargue in #4675
Execution
- Fix uploader missing events in block data uploaded to GCP by @fxamacker in #4562
- Use CCF in self-describing mode to encode events (replaces JSON-CDC) by @fxamacker in #4417
- panic when on executed block error by @zhangchiqing in #4464
- add execution scripts engine by @zhangchiqing in #4466
- Temporary workaround for LN networking issue by @peterargue in #3860
- Check if block exists locally in RPC endpoints by @peterargue in #4488
- Ingestion Engine - CollectionFetcher by @zhangchiqing in #4713
- Log stop height by @zhangchiqing in #4786
- Move the DefaultIndex from Start method to constructor by @zhangchiqing in #4843
- Refactor provider broadcast by @zhangchiqing in #4744
- Move ledger key to register id conversions by @janezpodhostnik in #4766
- Refactor ingestion test cases by @zhangchiqing in #4758
- read trie root hash from checkpoint by @zhangchiqing in #4811
- update ledger.Set to remove empty update case by @zhangchiqing in #4837
- Extract BlockLoader from execution Ingestion engine by @zhangchiqing in #4718
- log num txns and collections by @zhangchiqing in #4716
- log num txs and state changed in block executed log by @zhangchiqing in #4683
- clean up ingestion engine by removing unused dependencies by @zhangchiqing in #4689
- Add cmd checkpoint trie stats by @zhangchiqing in #4636
- adjust block execution time buckets by @zhangchiqing in #4505
- store chunk data pack in a separate database by @zhangchiqing in #4618
- add comments to explain why storage proof needs all registers ids by @zhangchiqing in #4486
- add export report by @zhangchiqing in #4427
FVM
- Use safe randomness from the distributed Random Beacon by @tarakby in #4498
- enable
unsafeRandom
for script execution by @tarakby in #4604 - use safe randomness for script execution in Cadence's
unsafeRandom
by @tarakby in #4648 - Upgrade cadence and change to ReadRandom by @janezpodhostnik in #4679
- Fix Interpreter Issue on Concurrent TX Execution by @janezpodhostnik in #4669
- port - Fix system transaction imports (#4443) by @janezpodhostnik in #4445
- port - Transaction logging fix by @janezpodhostnik in #4446
- Fix Version Beacon event conversion by @janezpodhostnik in #4546
- Remove dual authorizers for system transaction by @janezpodhostnik in #4554
- port - Dependency tracking on load error #4474 by @janezpodhostnik in #4475
- Add RandomBeaconHistory contract to system transaction #4582. by [@janezpodhostnik] and [@tarakby] in #4582
Cadence
- Update to Cadence v0.39.2 by @turbolent in #4434
- Update to Cadence v0.39.3 by @turbolent in #4438
- Update to Cadence v0.39.12 by @turbolent in #4507
- Update to onflow/cadence v0.39.14 by @SupunS in #4550
- Update to Cadence v0.39.8 by @turbolent in #4468
- Auto update to onflow/cadence v0.40.0 by @turbolent in #4603
- Update to Cadence v0.41.1 by @turbolent in #4733
- Auto update to onflow/cadence v0.42.0 by @turbolent in #4802
- Auto update to onflow/cadence v0.42.1 by @turbolent in #4867
HeroCache
- unifies method receivers by @yhassanzadeh13 in #4510
Localnet
- Remove unused hotstuff-proposal-time flag from bootstrap by @peterargue in #4481
- Fix semver for localnet builds by @peterargue in #4591
- Update metrics and logging images to latest versions by @peterargue in #4672
Metrics
- Added extra metrics for hotstuff data structures by @durkmurder in #4311
- Fix issue with collecting REST Metrics by @Guitarheroua in #4452
- Set correct value for
TotalComputationUsed
metric inChunk
by @m-Peter in #4577
Networking
- Fix chunk datapack size and timeout limits by @peterargue in #4623
- Fixes the bug with chunk data pack max message size. by @yhassanzadeh13 in #4530
- Sets GossipSub RPC Inspector mandatory default by @yhassanzadeh13 in #4511
- Upgrades libp2p version by @yhassanzadeh13 in #4537
- Handling iHave overpromising part-1 by @yhassanzadeh13 in #4556
- Handling iHave broken promises (and part-2 of overpromising) spams by @yhassanzadeh13 in #4566
- Relocate the logic for creating the spork ID within the test code by @yhassanzadeh13 in #4627
- Refinement of Test Fixtures: Isolating Tag Observable Logic in LibP2P Testing by @yhassanzadeh13 in #4629
- Fixes Flakey
TestFullGossipSubConnectivityAmongHonestNodesWithMaliciousMajority
by @yhassanzadeh13 in #4647 - Refinement of message scope; encapsulating the publish logic in the libp2p node by @yhassanzadeh13 in #4635
- GossipSub iWant Flooding Mitigation by @kc1116 in #4574
- Tiny refactoring of p2pNode by @nozim in #4630
- Refactoring Networking Layer for Improved Structure and Maintainability AND Eliminating Redundant Middleware Component by @yhassanzadeh13 in #4664
- Replaces CPU intensive
peer.ID
for libp2p with cache-based logging by @yhassanzadeh13 in #4702 - Control message validation unit test coverage upgrade by @kc1116 in #4695
- Optimizing Metric Cardinality and Easing Warning Score Criteria by @yhassanzadeh13 in #4790
- Isolates DHT capability within ANs and ENs by @yhassanzadeh13 in #4800
- Mitigates invalid message delivery by @yhassanzadeh13 in #4492
- Adaptive attempts on unicast dialing and stream creation by @yhassanzadeh13 in #4743
- Tuning libp2p resource manager limits; supporting by tests by @yhassanzadeh13 in #4846
-
- Move libP2P logs to debug level by @zhangchiqing in #4736
Light node
- Remove unused duplicated apiproxy code by @peterargue in #4440
- Enable REST API on observers by @UlyanaAndrukhiv in #4499
RandomBeacon
Script execution on Access node
- Create bootstrap checkpoint by @sideninja in #4759
- Script execution module by @sideninja in #4747
- use branching logic for Archive and Execution node script execution by @koko1123 in #4244
- Create validation mode for script exec on RN/EN by @koko1123 in #4573
- Script execution engine interface changes refactor by @sideninja in #4685
Execution data sync
- Add Execution Data ID Verification by @peterargue in #4586
- Add Tx Results to execution data by @peterargue in #4699
- Update ExecutionData rpc converters with tx results by @peterargue in #4710
- Create light tx result storage by @peterargue in #4735
- Update chunk execution data serializer header version by @peterargue in #4722
Storage
- Refactor badger cache and all the client code to use generics by @nozim in #4565
- Create base pebble storage by @koko1123 in #4671
- Remove outdated benchstat test by @koko1123 in #4711
- add pebble db init by @zhangchiqing in #4771
Testing & CI
- Refactoring Network Test Fixtures for Enhanced Customizability and Maintenance by @yhassanzadeh13 in #4524
- Split epoch integration tests into 2 cohorts by @jordanschalm in #4626
- Unquarantine TestMultiCluster (integration/tests/collection/proposal_test.go) by @gomisha in #4459
- Unquarantine TestExecutionStateSync (integration/tests/access/execution_state_sync_test.go) by @gomisha in #4460
- Unquarantine test - TestCrosstalkPreventionOnNetworkKeyChange by @gomisha in #4515
- fix verify-mocks intermittent CI failure by @gomisha in #4484
- Rename without_netgo tag to without-netgo by @peterargue in #4548
- Print builds and tools job inputs by @peterargue in #4617
- Update the workflows to use new auth by @peterargue in #4724
- Fix Flaky Test Monitor in CI - replace
run-tests.sh
bash script with GitHub Actions by @gomisha in #4589 - Split out
network
unit tests in CI by @gomisha in #4680 - Add merge_group to CI to allow using built in GH merge queue work flow by @Kay-Zee in #4607
Verification
- Improves logging details for chunk faults by @yhassanzadeh13 in #4509
- Refactor chunkVerifier unit tests by @peterargue in #4698
Crypto
- Improvements to BLS batch verify by @tarakby in #4225
- Update Go version to 1.20 by @tarakby in #4563
Protocol Data
- Fix last finalized header cache by @zhangchiqing in #4449
Chores & Misc other
- update code owners by @pattyshack in #4500
- Update workflow to add summary for GCS location for tools by @sjonpaulbrown in #4169
- Remove pinned docker tag from being created during builds by @sjonpaulbrown in #4442
- Update BN2 automation to support configuration uploads by @sjonpaulbrown in #4767
- Fix minor typos by @omahs in #4662
- Replaced Fork-Aware_Mempools.md broken link. by @Guitarheroua in #4737
- Removing 0 index as an undefined. by @kozlovb in #4470
- Update bootstrapping to include nft and metadata views by @joshuahannan in #4536
- Adding an array of state lists by @kozlovb in #4599
scriptExecutor
should populate env values even when the script errors by @m-Peter in #4572- Change queue.String() to iterate through children by @Kay-Zee in #4650
- Add a NO-OP implementation of
ReadRandom
onfacadeEnvironment
by @m-Peter in #4678 - Add version cmd to utils by @janezpodhostnik in #4614
- Pass zerolog by value by @janezpodhostnik in #4644
- Git tag validation by @janezpodhostnik in #4552
- Upgrade Emulator by @janezpodhostnik in #4575
- Make tests less flaky by @janezpodhostnik in #4581
New Contributors
- @joshuahannan made their first contribution in #4536
- @UlyanaAndrukhiv made their first contribution in #4411
- @nozim made their first contribution in #4565
- @kozlovb made their first contribution in #4470
- @omahs made their first contribution in #4662
- @AndriiDiachuk made their first contribution in #4812
Full Changelog: v0.31.21...v0.32.3
What's Changed
- [Access] 4215 - Add heartbeat responses to event streaming API by @AndriiDiachuk in #4812
- [Storehouse] [Ledger] update ledger.Set to remove empty update case by @zhangchiqing in #4837
- [ALSP] Synchronization Engine spam detection flag support,
config/README.md
updates by @gomisha in #4842 - [JobQueue] Move the DefaultIndex from Start method to constructor by @zhangchiqing in #4843
- Auto update to onflow/cadence v0.42.1 by @turbolent in #4867
- [Networking] Tuning libp2p resource manager limits; supporting by tests by @yhassanzadeh13 in #4846
New Contributors
- @AndriiDiachuk made their first contribution in #4812
Full Changelog: v0.32.2...v0.32.3
What's Changed
- [Access] 4215 - Add heartbeat responses to event streaming API by @AndriiDiachuk in #4812
- [Storehouse] [Ledger] update ledger.Set to remove empty update case by @zhangchiqing in #4837
- [ALSP] Synchronization Engine spam detection flag support,
config/README.md
updates by @gomisha in #4842 - [JobQueue] Move the DefaultIndex from Start method to constructor by @zhangchiqing in #4843
- Auto update to onflow/cadence v0.42.1 by @turbolent in #4867
- [Networking] Tuning libp2p resource manager limits; supporting by tests by @yhassanzadeh13 in #4846
New Contributors
- @AndriiDiachuk made their first contribution in #4812
Full Changelog: v0.32.2...v0.32.3
What's Changed
- [Access] 4215 - Add heartbeat responses to event streaming API by @AndriiDiachuk in #4812
- [Storehouse] [Ledger] update ledger.Set to remove empty update case by @zhangchiqing in #4837
- [ALSP] Synchronization Engine spam detection flag support,
config/README.md
updates by @gomisha in #4842 - [JobQueue] Move the DefaultIndex from Start method to constructor by @zhangchiqing in #4843
- Auto update to onflow/cadence v0.42.1 by @turbolent in #4867
- [Networking] Tuning libp2p resource manager limits; supporting by tests by @yhassanzadeh13 in #4846
New Contributors
- @AndriiDiachuk made their first contribution in #4812
Full Changelog: v0.32.2...v0.32.3
What's Changed
- [Access] 4215 - Add heartbeat responses to event streaming API by @AndriiDiachuk in #4812
- [Storehouse] [Ledger] update ledger.Set to remove empty update case by @zhangchiqing in #4837
- [ALSP] Synchronization Engine spam detection flag support,
config/README.md
updates by @gomisha in #4842 - [JobQueue] Move the DefaultIndex from Start method to constructor by @zhangchiqing in #4843
- Auto update to onflow/cadence v0.42.1 by @turbolent in #4867
- [Networking] Tuning libp2p resource manager limits; supporting by tests by @yhassanzadeh13 in #4846
New Contributors
- @AndriiDiachuk made their first contribution in #4812
Full Changelog: v0.32.2...v0.32.3
What's Changed
- [Access] 4215 - Add heartbeat responses to event streaming API by @AndriiDiachuk in #4812
- [Storehouse] [Ledger] update ledger.Set to remove empty update case by @zhangchiqing in #4837
- [ALSP] Synchronization Engine spam detection flag support,
config/README.md
updates by @gomisha in #4842 - [JobQueue] Move the DefaultIndex from Start method to constructor by @zhangchiqing in #4843
- Auto update to onflow/cadence v0.42.1 by @turbolent in #4867
- [Networking] Tuning libp2p resource manager limits; supporting by tests by @yhassanzadeh13 in #4846
New Contributors
- @AndriiDiachuk made their first contribution in #4812
Full Changelog: v0.32.2...v0.32.3