v1.28.0

[romange]

Dragonfly v1.28.0 - Spring Edition 💘

This marks our second release for the year, with lots of bugs closed and probably some new bugs introduced.

Important fixes and changes:

1. Deadlocked transaction bugs fix(server): list move on single shard wake blocking transaction #4590 server: disable single shard tx optimization on scheduling #4647 fix(server): deadlock with replicaof inside multi #4685
2. Sharded pubsub (API-7) : feat(server): Add PUBSUB SHARDCHANNELS/SHARDNUMSUB #4702 chore: implement sharded pub/sub #4518 chore: unsubscribe pub/sub connections after cluster migration #4529
3. Fix Streams replication fix(stream_family): Fix replication for the XADD and XTRIM commands #4591 fix(transaction): Fix auto journaling in transaction #4737
4. Numerous cluster improvements feat: allow cluster READONLY command #4727 feat(server): Cluster MOVED response Prometheus metric #4786 feat: add node health status for CLUSTER SLOTS and SHARDS #4767

What's Changed

• fix(search_family): Add options test for the FT.AGGREGATE command by @BagritsevichStepan in fix(search_family): Add options test for the FT.AGGREGATE command #4479
• fix(stream_family): Fix replication for the XADD and XTRIM commands by @BagritsevichStepan in fix(stream_family): Fix replication for the XADD and XTRIM commands #4591
• chore(search_family): Simplify FT.SEARCH reply code by @BagritsevichStepan in chore(search_family): Simplify FT.SEARCH reply code #4553
• feat(zset family): Implement ZDIFFSTORE by @mkaruza in feat(zset family): Implement ZDIFFSTORE #4582
• fix(seeder): Support stream types in the Seeder by @BagritsevichStepan in fix(seeder): Support stream types in the Seeder #4574
• fix: debian path in dragonfly.service by @romange in fix: debian path in dragonfly.service #4594
• fix: tls replication without ca verification by @kostasrim in fix: tls replication without ca verification #4539
• fix(server): list move on single shard wake blocking transaction by @adiholden in fix(server): list move on single shard wake blocking transaction #4590
• fix: race when bumping items while loading a snapshot by @kostasrim in fix: race when bumping items while loading a snapshot #4564
• fix: multi/exec CROSSSLOT error processing by @BorysTheDev in fix: multi/exec CROSSSLOT error processing #4598
• test: added test_keys_expiration_during_migration by @BorysTheDev in test: added test_keys_expiration_during_migration #4558
• chore: fix test_snapshot by @kostasrim in chore: fix test_snapshot #4607
• chore: fix glob translation with \ at the end by @romange in chore: fix glob translation with \ at the end #4608
• feat: support lz4 compression method by @romange in feat: support lz4 compression method #4610
• fix: test_keys_expiration_during_migration by @BorysTheDev in fix: test_keys_expiration_during_migration #4613
• chore: add zstd_test by @romange in chore: add zstd_test #4614
• chore: simplify transaction code by @romange in chore: simplify transaction code #4602
• fix: epoll reg tests running on iouring by @kostasrim in fix: epoll reg tests running on iouring #4612
• chore: use macos-14 for the daily macos build by @kostasrim in chore: use macos-14 for the daily macos build #4609
• chore: introduce docker release pipeline by @romange in chore: introduce docker release pipeline #4618
• chore: connection pipeline cache does not shrink by @kostasrim in chore: connection pipeline cache does not shrink #4491
• chore: implement sharded pub/sub by @kostasrim in chore: implement sharded pub/sub #4518
• chore: introduce background deletions of DenseSet objects by @romange in chore: introduce background deletions of DenseSet objects #4496
• fix: revert reflex based matching due to performance problems by @romange in fix: revert reflex based matching due to performance problems #4626
• chore: allow yielding in Scan command by @kostasrim in chore: allow yielding in Scan command #4624
• refactor: simplify journal and restore streamer cancellation by @BorysTheDev in refactor: simplify journal and restore streamer cancellation #4549
• server: rdb loader big string loading in chunks by @adiholden in server: rdb loader big string loading in chunks #4623
• chore: remove redundant calls to GetSlice by @romange in chore: remove redundant calls to GetSlice #4635
• server: redis parser reject big bulk read by @adiholden in server: redis parser reject big bulk read #4627
• chore: Improve loading of sorted sets by @romange in chore: Improve loading of sorted sets #4631
• fix: invalid BUSYKEY response by @romange in fix: invalid BUSYKEY response #4637
• chore: propagate reserve data upon restore by @romange in chore: propagate reserve data upon restore #4641
• chore: get rid of sds in SortedMap::AddElem by @romange in chore: get rid of sds in SortedMap::AddElem #4638
• fix: improve test_send_delay_metric test by @romange in fix: improve test_send_delay_metric test #4642
• fix: do not flush replicaof if failed to connect by @kostasrim in fix: do not flush replicaof if failed to connect #4636
• fix: performance degradation with Reflex matching by @romange in fix: performance degradation with Reflex matching #4634
• chore: get rid of lispack stats by @romange in chore: get rid of lispack stats #4644
• chore: add listpack statistics to objhist by @romange in chore: add listpack statistics to objhist #4650
• chore: allow sampling of topk hottest keys. by @romange in chore: allow sampling of topk hottest keys. #4649
• feat: implement unique keys counting using hll by @romange in feat: implement unique keys counting using hll #4652
• fix: pull latest helio by @romange in fix: pull latest helio #4651
• chore: unsubscribe pub/sub connections after cluster migration by @kostasrim in chore: unsubscribe pub/sub connections after cluster migration #4529
• fix: bring back the helm step to docker release action by @romange in fix: bring back the helm step to docker release action #4657
• fix: do not crash upon inconsistent expire iterator by @romange in fix: do not crash upon inconsistent expire iterator #4658
• chore: replace try assert false with pytest.raises by @kostasrim in chore: replace try assert false with pytest.raises #4662
• server: disable single shard tx optimization on scheduling by @adiholden in server: disable single shard tx optimization on scheduling #4647
• fix: race condition in ReplicaOfInternal by @kostasrim in fix: race condition in ReplicaOfInternal #4653
• Support Cluster configuration for dfly_bench by @romange in Support Cluster configuration for dfly_bench #4664
• CI: add slow regression tests by @adiholden in CI: add slow regression tests #4646
• fix: replication error message and scan command yield by @kostasrim in fix: replication error message and scan command yield #4665
• feat: add migration error statistic by @BorysTheDev in feat: add migration error statistic #4643
• chore: refactor ServerFamily::Info by @romange in chore: refactor ServerFamily::Info #4671
• fix(search_family): Add option tests for the FT.CREATE command by @BagritsevichStepan in fix(search_family): Add option tests for the FT.CREATE command #4678
• chore: introduce DEBUG COMPRESSION by @romange in chore: introduce DEBUG COMPRESSION #4620
• chore: cluster related clean ups by @romange in chore: cluster related clean ups #4683
• fix: SendInvalidationTrackingMessage should not block. by @romange in fix: SendInvalidationTrackingMessage should not block. #4680
• fix: reduce stack usage on Fedora by @romange in fix: reduce stack usage on Fedora #4690
• fix(server): deadlock with replicaof inside multi by @adiholden in fix(server): deadlock with replicaof inside multi #4685
• fix(metrics): add corresponding comments to command metrics by @Yomanz in fix(metrics): add corresponding comments to command metrics #4674
• feat: Added a flag to ignore key expiry. by @H4R5H1T-007 in feat: Added a flag to ignore key expiry. #4667
• feat(cluster_mgr): Add argument to set path to dragonfly binary by @mkaruza in feat(cluster_mgr): Add argument to set path to dragonfly binary #4695
• fix: possible preemption under FiberAtomicGuard by @romange in fix: possible preemption under FiberAtomicGuard #4692
• chore: add debug printings to SetExisting by @romange in chore: add debug printings to SetExisting #4694
• chore: Add debug logs to help tracking transactional deadlocks by @romange in chore: Add debug logs to help tracking transactional deadlocks #4669
• chore: snapshot/tiering improvements by @romange in chore: snapshot/tiering improvements #4661
• chore: disable failing epoll tests by @kostasrim in chore: disable failing epoll tests #4699
• feat: client unpause and client help by @kostasrim in feat: client unpause and client help #4630
• fix: ZMSCORE return value if key does not exist by @vyavdoshenko in fix: ZMSCORE return value if key does not exist #4697
• fix: prevent tx_executor reading from sock if operation is canceled by @BorysTheDev in fix: prevent tx_executor reading from sock if operation is canceled #4704
• feat(set_family): Update TTL for existing fields in SADDEX by @abhijat in feat(set_family): Update TTL for existing fields in SADDEX #4700
• chore: update CONTRIBUTING.md by @kostasrim in chore: update CONTRIBUTING.md #4703
• chore: minor clean ups before introducing ProvidedBuffers by @romange in chore: minor clean ups before introducing ProvidedBuffers #4709
• refactor: clean cluster code by @BorysTheDev in refactor: clean cluster code #4707
• fix: Remove the dfly logo print from pytest run Remove the dfly logo print from pytest run #4682 by @vyavdoshenko in fix: Remove the dfly logo print from pytest run #4682 #4706
• fix(json_family): Fix JSON.SET handling for nested fields by @BagritsevichStepan in fix(json_family): Fix JSON.SET handling for nested fields #4710
• fix(dfly_bench): support dns resolution for cluster hosts by @romange in fix(dfly_bench): support dns resolution for cluster hosts #4715
• fix: buffer overrun in GetRandomHex by @romange in fix: buffer overrun in GetRandomHex #4717
• feat(set_family): Add support for KEEPTTL to SAddEx by @abhijat in feat(set_family): Add support for KEEPTTL to SAddEx #4712
• fix: preemption in atomic section of heartbeat by @kostasrim in fix: preemption in atomic section of heartbeat #4720
• Revert "chore: minor clean ups before introducing ProvidedBuffers (add missing options in already implemented commands that appear in 2.8 #4… by @adiholden in Revert "chore: minor clean ups before introducing ProvidedBuffers (#4… #4719
• feat(server): Add PUBSUB SHARDCHANNELS/SHARDNUMSUB by @mkaruza in feat(server): Add PUBSUB SHARDCHANNELS/SHARDNUMSUB #4702
• fix(dfly_bench): track hit rate for mget command by @romange in fix(dfly_bench): track hit rate for mget command #4723
• refactor: move tl_cluster_config into cluster_config.cc by @BorysTheDev in refactor: move tl_cluster_config into cluster_config.cc #4714
• feat: allow cluster READONLY command by @BorysTheDev in feat: allow cluster READONLY command #4727
• fix: improve stack margin for s3 related operations. by @romange in fix: improve stack margin for s3 related operations. #4731
• chore: Add clion build path to gitignore by @abhijat in chore: Add clion build path to gitignore #4732
• server: control cluster migration speed with flag by @adiholden in server: control cluster migration speed with flag #4734
• chore: improve debug compression by @romange in chore: improve debug compression #4735
• feat: dfly acl select by @kostasrim in feat: dfly acl select #4696
• feat: add an option to start using iouring provided buffers by @romange in feat: add an option to start using iouring provided buffers #4736
• fix: enforce strong consistency in channel_store by @kostasrim in fix: enforce strong consistency in channel_store #4740
• fix(transaction): Fix auto journaling in transaction by @BagritsevichStepan in fix(transaction): Fix auto journaling in transaction #4737
• feat(hset_family): Add KEEPTTL support to HSetEx by @abhijat in feat(hset_family): Add KEEPTTL support to HSetEx #4730
• chore: replace chrono with absl in PipelineCacheSizeTracker by @kostasrim in chore: replace chrono with absl in PipelineCacheSizeTracker #4748
• feat: allow non-ascii values in dfly_bench by @romange in feat: allow non-ascii values in dfly_bench #4751
• feat(server): Support dynamic SAVE path by @mkaruza in feat(server): Support dynamic SAVE path #4729
• feat(server): extend populate command with expire by @lichuang in feat(server): extend populate command with expire #4752
• chore: Handle hsetex options in any order by @abhijat in chore: Handle hsetex options in any order #4754
• fix test: test_replication_all increase key num by @adiholden in fix test: test_replication_all increase key num #4747
• fix(server): hmget non uniqe keys response by @adiholden in fix(server): hmget non uniqe keys response #4745
• feat(metrics): Add label for main and other listeners by @abhijat in feat(metrics): Add label for main and other listeners #4739
• feat(server): extend populate command with expire, fix bug that expire the same key for multiple time by @lichuang in feat(server): extend populate command with expire, fix bug that expire the same key for multiple time #4757
• feat: add node health status into config by @BorysTheDev in feat: add node health status into config #4758
• fix(test): Execute SSUBSCRIBE command directly in client by @mkaruza in fix(test): Execute SSUBSCRIBE command directly in client #4763
• fix: optional brackets were removed for ipv6 host by @vyavdoshenko in fix: optional brackets were removed for ipv6 host #4759
• chore: clean up dbslice by @kostasrim in chore: clean up dbslice #4769
• fix: proper memory accounting for objects loaded via streaming by @romange in fix: proper memory accounting for objects loaded via streaming #4774
• fix bug : moved error port number on migration finish by @adiholden in fix bug : moved error port number on migration finish #4776
• feat: add node health status for CLUSTER SLOTS and SHARDS by @BorysTheDev in feat: add node health status for CLUSTER SLOTS and SHARDS #4767
• fix: removal of Expiry bit when overriding external strings by @romange in fix: removal of Expiry bit when overriding external strings #4785
• Fix extended node info comparison by @BorysTheDev in Fix extended node info comparison #4788
• feat(server): Cluster MOVED response Prometheus metric by @mkaruza in feat(server): Cluster MOVED response Prometheus metric #4786
• fix: invalid reference inside GetQueueBackpressure by @romange in fix: invalid reference inside GetQueueBackpressure #4789

Huge thanks to all the contributors! ❤️

New Contributors

• @mkaruza made their first contribution in feat(zset family): Implement ZDIFFSTORE #4582
• @Yomanz made their first contribution in fix(metrics): add corresponding comments to command metrics #4674
• @H4R5H1T-007 made their first contribution in feat: Added a flag to ignore key expiry. #4667
• @abhijat made their first contribution in feat(set_family): Update TTL for existing fields in SADDEX #4700
• @lichuang made their first contribution in feat(server): extend populate command with expire #4752

Full Changelog: v1.27.0...v1.28.0

---

This discussion was created from the release v1.28.0.