v0.20.4-test2

fix(goreleaser): tag_sort (#3972)

v0.20.4-test1

Test release

v0.20.4

This is a security patch release, please upgrade as soon as possible.

What's Changed

• refactor(share): GetShare -> GetSamples by @Wondertan in #3891
• deps(go.mod): bump app by @renaynay in #3968
• fix: jwt token nonce and expiration time by @cristaloleg in #3967
• chore(deps): upgrade to celestia-app v3.0.2 by @rootulp in #3969

Full Changelog: v0.20.3...v0.20.4

v0.20.4-mocha

This is a security patch release for the mocha network, please upgrade as soon as possible.

What's Changed

• fix(openrpc-gen): add go-square blob to examples by @vgonkivs in #3962
• fix(.github/workflows): no automatic tagging of releases for CI by @renaynay in #3963
• refactor(share): GetShare -> GetSamples by @Wondertan in #3891
• deps(go.mod): bump app by @renaynay in #3968
• fix: jwt token nonce and expiration time by @cristaloleg in #3967
• chore(deps): upgrade to celestia-app v3.0.2 by @rootulp in #3969

Full Changelog: v0.20.3-mocha...v0.20.4-mocha

v0.20.4-arabica

This is a security patch release for the arabica network, please upgrade as soon as possible.

What's Changed

• fix(openrpc-gen): add go-square blob to examples by @vgonkivs in #3962
• fix(.github/workflows): no automatic tagging of releases for CI by @renaynay in #3963
• refactor(share): GetShare -> GetSamples by @Wondertan in #3891
• deps(go.mod): bump app by @renaynay in #3968
• fix: jwt token nonce and expiration time by @cristaloleg in #3967
• chore(deps): upgrade to celestia-app v3.0.2 by @rootulp in #3969

Full Changelog: v0.20.3-arabica...v0.20.4-arabica

v0.20.3

This patch release fixes a JSON serialisation error with libshare.Blob via #3958, fixing the state.SubmitPayForBlob endpoint.

What's Changed

• chore(deps): Bump cosmossdk.io/math from 1.3.0 to 1.4.0 by @dependabot in #3955
• chore(go.mod) bump go-square by @vgonkivs in #3958
• fix(openrpc-gen): add go-square blob to examples by @vgonkivs in #3962
• fix(.github/workflows): no automatic tagging of releases for CI by @renaynay in #3963

Full Changelog: v0.20.2...v0.20.3

v0.20.3-mocha

This patch release fixes a JSON serialisation error with libshare.Blob via #3958.

What's Changed

• refactor: remove blob size check by @rootulp in #3934
• fix(core): change fill block mode by @cristaloleg in #3936
• hotfix(core): don't hold on Data by @Wondertan in #3926
• fix(p2p): disable quic by @walldiss in #3937
• chore(deps): temporary fork boxo by @Wondertan in #3939
• refactor(share/availability): Move window and constants to share/availability pkg by @renaynay in #3906
• fix(shwap/bitswap): tune bitswap params by @Wondertan in #3940
• fix(shwap/bitswap): Blockstore.GetSize: getting size with no compute by @Wondertan in #3894
• chore(logs): cleanup logging by @Wondertan in #3945
• fix(shwap/bitswap): update boxo and config by @Wondertan in #3946
• chore(go.mod): bump boxo fork by @Wondertan in #3948
• fix(share/availability/light): Allow retries on ODS size 1 by @renaynay in #3944
• perf(shwap/bitswap): add session pools inside Getter by @Wondertan in #3947
• chore(deps): Bump cosmossdk.io/math from 1.3.0 to 1.4.0 by @dependabot in #3955
• chore(go.mod) bump go-square by @vgonkivs in #3958

Full Changelog: v0.20.2-mocha...v0.20.3-mocha

v0.20.3-arabica

This patch release fixes a JSON serialisation error with libshare.Blob via #3958.

What's Changed

• refactor: remove blob size check by @rootulp in #3934
• fix(core): change fill block mode by @cristaloleg in #3936
• hotfix(core): don't hold on Data by @Wondertan in #3926
• fix(p2p): disable quic by @walldiss in #3937
• chore(deps): temporary fork boxo by @Wondertan in #3939
• refactor(share/availability): Move window and constants to share/availability pkg by @renaynay in #3906
• fix(shwap/bitswap): tune bitswap params by @Wondertan in #3940
• fix(shwap/bitswap): Blockstore.GetSize: getting size with no compute by @Wondertan in #3894
• chore(logs): cleanup logging by @Wondertan in #3945
• fix(shwap/bitswap): update boxo and config by @Wondertan in #3946
• chore(go.mod): bump boxo fork by @Wondertan in #3948
• fix(share/availability/light): Allow retries on ODS size 1 by @renaynay in #3944
• perf(shwap/bitswap): add session pools inside Getter by @Wondertan in #3947
• chore(deps): Bump cosmossdk.io/math from 1.3.0 to 1.4.0 by @dependabot in #3955
• chore(go.mod) bump go-square by @vgonkivs in #3958

Full Changelog: v0.20.2-arabica...v0.20.3-arabica

v0.20.2

NOTE: v0.18 and v0.19 are skipped for mainnet

Hey everyone, this is quite a mammoth of a release, so please hang tight. We have two major changes packaged in this release:

• Shwap (protocol-breaking)
• Celestia v3 compatibility (API-breaking)

The migration period for this release is quite tight (a bit longer than 2 weeks), so we urge all node operators to upgrade to this version and re-sync against the network by December 2nd.

As this release is API-breaking, we also urge all integrations to migrate to this new version by December 2nd. (The list of API breaks can be found below).

Shwap

This release features Shwap(CIP-19), a protocol-breaking feature that brings massive optimizations to disk usage(16x) and data access latencies. The new protocol is not backwards compatible with the currently deployed protocol and software.

Both protocols will run side-by-side for about a two-week transition period, from November 20 to December 2. During this period, node operators should migrate to the new version. At the end of the period, posted historical and recent data will no longer be available on the old protocol.

Migration Notes

Migration requires resyncing the node from scratch, as it now contains a new storage format.

Migration Steps over the old node:

• Remove old data:
  • In home directory(default .celestia-<type>) => rm -rf inverted_index index data transients blocks
  • Keep keys directory to preserve peer and chain identity keys in place
• Config updates
  • If defaults were in use, remove the old config and re-init in the next step
  • If you want to preserve the old config - move BlockstoreCacheSize entry from [Share.EDSStoreParams] section to just [Share]and remove GCInterval, RoutingTableRefreshPeriod
• Re-init node for the Mocha p2p network
  • celestia init
• Start the node and monitor syncing
  • celestia start

If the old node has to be kept running - initiate and start the new version in a different directory:

• celestia init --node.store <new_path>
• celestia start --node.store <new_path>

Celestia V3 compatibility

This release contains an upgrade to the celestia-app version (to v3). v3 will only be live on the mainnet in mid-December, but this release contains the requirements for compatibility with the upgrade.

Notable API breaks include:

• simplification of the blob share and state modules via the deduplication of types (via [#3782](#3782))
• share module now takes height instead of the full ExtendedHeader (via [#3870](#3870))

Comprehensive list of API breaks:

• GetSharesByNamespace renamed to GetNamespaceData
• the share module returns shwap.NamespaceData for the GetNamespaceData method
• the share module now takes height (uint64) instead of the full ExtendedHeader for all methods
• both the share and blob modules now use the go-square v2 libshare.Namespace (from https://github.com/celestiaorg/go-square/tree/v2.0.0) instead of the old share.Namespace for all methods
• the state module now uses the go-square v2 libshare.Blob type (from https://github.com/celestiaorg/go-square/tree/v2.0.0) instead of the old one from the state module
• the blob package now embeds the new go-square v2 libshare.Blob type mentioned above in the blob.Blob struct instead of the old v1 type.

Miscellaneous

What to look out for in the following releases:

• the -experimental-pruning flag will soon be deprecated in favour of pruning becoming the default mode for all node types, requiring users to pass -archival to retain block history beyond the sampling window
• the API gateway will be deprecated

What's Changed

Features

• feat(shwap): Add eds store by @walldiss in #3545
• feat(shwap): Add namespace data shwap id by @walldiss in #3587
• feat(shwap): Add axis roots to Accessor by @walldiss in #3586
• feat(shwap): Add bufferisation for writing ods/Q1Q4 files by @walldiss in #3592
• feat(shwap/p2p): Bitswap Getter by @Wondertan in #3584
• feat(shwap): Shwap add readFrom Writeto methods to shwap ids by @walldiss in #3605
• feat(shwap): Integrate shrex into shwap by @walldiss in #3554
• feat(shwap): Shwap core integration by @walldiss in #3598
• feat(shwap): Shwap cascade and mock getters by @walldiss in #3599
• feat(shwap): Update availability to work with shwap by @walldiss in #3600
• feat(shwap): add shwap to blob module by @walldiss in #3613
• feat(shwap): update daser for shwap by @walldiss in #3614
• feat(shwap): add shwap store getter by @walldiss in #3617
• feat(shwap): Add caching to blockstore by @walldiss in #3615
• feat(shwap/bitswap): Shwap optimized Bitswap constructors by @Wondertan in #3536
• feat(shwap): Improve eds store cache refcounter forgot Close() protection by @walldiss in #3638
• feat(shwap): shwap nodebuilder and cleanup by @walldiss in #3637
• feat(shwap/ods_file): tail padding trim by @Wondertan in #3620
• feat(shwap): Add discovery version by @walldiss in #3616
• feat(cmd/cel-shed): shwap by @Wondertan in #3720
• feat(shwap/store): Add q4 trimming support in store by @walldiss in #3680
• feat(nodebuilder): Blockstore cache params by @walldiss in #3739
• feat(share/availability): Dont store q4 on archival by @walldiss in #3751
• feat(store/cache): add Has methods by @Wondertan in #3757
• feat(cel-shed/p2p): bootstrapper probe tool by @smuu in #3780
• feat!: Live, laugh, Shwap by @Wondertan in #3675
• feat(cmd): adding timeout flag by @sebasti810 in #3580
• feat(nodebuilder/da): implement go-da v0.6.1 changes by @tzdybal in #3750
• feat(modp2p): close peer connections after successful blocking by @notlelouch in #3838
• feat(shwap/bitswap): use new option for optimized Has check by @Wondertan in #3813
• feat(store): Test all put funcs by @walldiss in #3847
• feat(store): use use relative path for empty blocks symlinks by @walldiss in #3861
• feat(store): correct corrupted files on write by @walldiss in #3859
• feat(share/availability)!: Store success results for light availability by @walldiss in #3887
• feat(pruner/light): add light pruning for shwap by @walldiss in #3896

Fixes

• fix(shwap/bitswap): close acccessor in Blockstore by @Wondertan in #3535
• fix(shwap/store): ensure EDS is accessible during writes by @Wondertan in #3596
• fix(shwap): don't cache empty eds and close recent accessor by @walldiss in #3608
• fix(store/files): ODSQ4 Closing audit by @Wondertan in #3691
• fix(shwap/cahce): use local copy of axis in proofs cache by @walldiss in #3657
• fix(shwap): io.ReadFull for read helpers by @Wondertan in #3696
• fix(shwap): io.ReadFull for NamespaceDataID by @walldiss in #3697
• fix(shwap/bitswap): correct protocol id by @Wondertan in #3722
• fix(store/odsq4): fix order of opened flag stored by @walldiss in #3735
• fix(store/ods): fix full ods caching by @walldiss in #3736
• fix(core): fix flaky generateNonEmptyBlocks by @walldiss in #3740
• fix(blob): fix blobsub subscription cancellation test by @walldiss in #3746
• fix(store): use symlinks for empty blocks heighs by @walldiss in #3747
• fix(header): soft failure is only for not enough voting power by @cristaloleg in #3769...

v0.20.2-mocha

This patch brings numerous fixes to the sampling protocol, particularly Bitswap sub-protocol dependency, by fixing 7(!) client-side memory leaks. It also improves resource utilization and load balancing on the server side. All the changes are currently applied in our fork(https://github.com/celestiaorg/boxo/tree/fixes) and will be upstreamed with unforking in an upcoming release.

Additionally, the patch applies a safety measure by temporarily disabling QUIC due to the unknown origin of the following bug: quic-go/quic-go#4712.

As a cherry on top, this patch hotfixes long-standing memory issues in BNs: #3926. Thanks to @rach-id for helping us with the debugging.

Full Changelog: v0.20.1-mocha...v0.20.2-mocha