1.3.0-rc0

What's Changed

• fix(s3stream): fix deadlock issue by @ShadowySpirits in #1808
• fix(e2e): adaptation to pr: #1794 by @llzcx in #1809
• feat(shell): new consumer in pref command should start consuming from the beginning by @ShadowySpirits in #1812
• fix(metadata): fix sparse index serialization capacity by @SCNieh in #1814
• chore(logger): move trim log to broker by @superhx in #1819
• chore(workflows): e2e support release runner by @superhx in #1825
• chore(config): change quorum default timeout to 5s by @superhx in #1823
• fix(s3strea/wal): handle IOException during flushing WAL header by @Chillax-0v0 in #1827
• chore(workflows): add e2e base image release by @superhx in #1830
• fix(test): fix rolling_update_test e2e by @superhx in #1835
• feat(s3stream): add AsyncLRUCache metric by @lifepuzzlefun in #1829
• fix(s3stream): fix storage failure handle deadlock by @superhx in #1838
• fix(s3stream/wal): fail all IO operations once the WAL is failed by @Chillax-0v0 in #1840
• feat(metadata): limit the size of sparse index cache by @SCNieh in #1832
• feat(issues1842): cleanup metastream kv after deleting topic by @superhx in #1843
• fix(s3stream): error callback by @superhx in #1845
• fix(revert): error callback by @superhx in #1851
• feat(metadata): avoid serialize empty index by @SCNieh in #1852
• feat(metadata): limit sparse index cache update interval by @SCNieh in #1847
• feat(s3stream): support call #close(force=true) after #close(force=false) by @superhx in #1854
• feat(metadata): prune invalid sparse index with image on startup by @SCNieh in #1856
• feat(s3stream): add network rate limiter for s3 wal to export metrics by @ShadowySpirits in #1859
• feat(s3stream): change the checksum algorithm to crc32c by @ShadowySpirits in #1860
• feat(s3stream): support change checksum algorithm by @ShadowySpirits in #1863
• feat(e2e): use cloud vms for e2e tests; move wal path by @mooc9988 in #1862
• fix(e2e): always archive result artifacts by @mooc9988 in #1866
• fix(storage): regard all produce requests as duplicate whose sequence… by @Chillax-0v0 in #1865
• revert: enable LocalStreamRangeIndexCacheTest#testEvict by @Chillax-0v0 in #1869
• feat(wal): failoverTest optimization by @CLFutureX in #1871
• fix(metadata): catch exception to prevent unnecessary error log by @SCNieh in #1872
• perf(s3stream): check the logger level before trace and debug by @Chillax-0v0 in #1875
• perf(s3stream/wal): reuse the ByteBuf for record headers by @Chillax-0v0 in #1877
• perf(s3stream/wal): increase the size of the buffer pool as CPU cores by @Chillax-0v0 in #1882
• chore(tools/perf): increase the message sending rate during the warmup to accelerate JVM warmup by @Chillax-0v0 in #1883
• test(metadata/image): thread sync issue by @Pey-crypto in #1879
• fix(test): fix consumer mode e2e by increase timeout by @superhx in #1885
• feat(issue1462): add user case. by @vintagewang in #1889
• fix(e2e): fix bug by @llzcx in #1891
• chore(logger): change e2e streams log level to INFO by @superhx in #1892
• perf(s3stream): optimize the critical section in s3 wal by @ShadowySpirits in #1880
• test(e2e): version upgrade test by @llzcx in #1822
• fix(s3stream/wal): fix memory leak by @Chillax-0v0 in #1896
• fix(e2e): fix data map format by @mooc9988 in #1898
• fix(metadata): fix sparse index leak on compaction split by @SCNieh in #1894
• fix(metadata): fix evict sparse index cache potential endless loop by @SCNieh in #1900
• feat(metadata): add metrics for sparse index cache monitor by @SCNieh in #1902
• fix(metadata): prevent concurrent uploading of local sparse index cache by @SCNieh in #1904
• fix(metadata): fix wrong stream number in serialization by @SCNieh in #1906
• feat(s3stream): add docs for checksumAlgorithm configuration by @ShadowySpirits in #1908
• fix(metadata): fix interrupted batch upload by @SCNieh in #1913
• perf(config): increase the default index interval from 4KiB to 1MiB by @Chillax-0v0 in #1915
• chore: add nightly release for automq by @daniel-y in #1916
• chore: polish the nightly release by @daniel-y in #1917
• chore: make the nightly release as a pre release by @daniel-y in #1918
• feat(s3stream): unify object not exist exception by @superhx in #1920
• fix(e2e): fix bug by @llzcx in #1922
• feat(s3stream): limit the inflight fast retry request count by @ShadowySpirits in #1919
• feat(eventLoop): the visibility of the shutdown flag for the event loop and optimizations for graceful shutdown by @CLFutureX in #1796
• chore(test): set runner to e2e by @superhx in #1929
• chore(telemetry): set the endpoint of otlp receiver manually by @Chillax-0v0 in #1927
• feat(metadata): expire node sparse index cache after write by @SCNieh in #1925
• feat(auto_balancer): limit the max number of nodes to reassign partit… by @SCNieh in #1933
• feat(metadata): blocking wait for uploading local index cache on stre… by @SCNieh in #1931
• feat(readme): add github trending logo by @vintagewang in #1936
• feat(s3stream): add checksum for s3 wal by @ShadowySpirits in #1939
• feat(s3stream): ensure compaction could be executed (#1935) by @superhx in #1937
• docs(s3stream/version): statement of features supported by S3Stream V1 by @Chillax-0v0 in #1943
• fix(s3stream): resolve unintended buffer reuse by @ShadowySpirits in #1945
• feat(s3stream): make s3client configuration can be override by @warr99 in #1947
• fix(s3stream): fix buffer leak by @ShadowySpirits in #1948
• fix(bufAlloc): adjust the default size by @CLFutureX in #1955
• perf(core): disable DeleteProduce tryComplete trace log by @lifepuzzlefun in #1951
• fix(s3stream): change reject handler to prevent task lost by @SCNieh in #1956
• feat(s3stream): change append callback thread number by @lifepuzzlefun in #1950
• feat(s3stream): support limit write bandwidth for WALBlockDeviceChannel by @lifepuzzlefun in #1941
• feat(readme): update aws marketplace url(#1936) by @vintagewang in #1958
• fix(issues1960): allow UNSET attributes for version 1.1 by @superhx in #1962
• build(dependencies): update Jackson to 2.17.1 by @warr99 in #1963
• feat(s3stream): add tail read throttle strategy by @SCNieh in #1965
• feat(wal): unOverCapacityException will lead to memory leaks by @CLFutureX in #1952
• perf(core): optimize FetchSession.update high memory allocate rate by @lifepuzzlefun in #1966
• perf(s3stream): optimize ByteBufAlloc.USAGE_STATS high cpu usage by @lifepuzzlefun in #1959
• fix(s3stream/wal): record metrics on WAL full by @Chillax-0v0 in #1968
• feat(s3stream): optimize recovery read throttle priority by @SCNieh in #1967
• Update README.md with new videos by @KaimingWan in #1969
• fix(log): use the same view to calculate trim offsets by @Chillax-0v0 in #1973
• fix(s3stream): fix namespace by @ShadowySpirits in #1977
• feat(core): default configuration optimization by @SCNieh in #1979
• chore(s3stream): log object id to identify the upload WAL request by @Chillax-0v0 in #1984
• feat(auto_balancer): limit the maximum time to execute actions by @SCNieh in #1989
• fix(auto_balancer): fix broker status change by @SCNieh in #1991
• fix(auto_balancer): fix string format by @SCNieh in #1987
• chore: beautify ElasticLogMeta in logs by @Chillax-0v0 in #1995
• feat(s3stream): increase compaction throttle priority by @SCNieh in #1994
• chore(util): implement IdURI#toString by @Chillax-0v0 in #1998
• fix(issues1999): fix Processor.channelContexts memory leak by @superhx in #2000
• fix(issue2004): fix AutoBalancerMetricsReporter cannot process with t… by @SCNieh in #2005
• fix(core): release all records before delayed fetch by @ShadowySpirits in #2008
• perf(DelayedFetch): only try to fast read on complete a delayed fetch by @Chillax-0v0 in #2014
• fix(core): rsp immediately for catch-up read even if rst is not enough by @SCNieh in #2018
• fix(core): remove offset metrics when expired by @SCNieh in #2023
• feat(metrics): skip reporting metrics delay for newly created partition by @SCNieh in #2028
• fix(metrics): fix potential NPE when exporting metrics by @SCNieh in #2029
• feat(docker): add docker shell for new version by @cuisongliu in #2027
• feat(api): add produce interceptor by @superhx in #2032
• fix(stream): fix compaction memory leak on shutdown by @SCNieh in #2033
• feat(s3stream): expose network limiter by @superhx in #2035
• refactor(s3stream/wal): use bucket4j to limit the write rate and bandwidth by @Chillax-0v0 in #2034
• perf(storage): increase the timeout to wait for the SEQ-0 in ProducerStateManager by @Chillax-0v0 in #2031
• feat(produce): add more info to client id by @superhx in #2036
• feat(zonerouter): add client type by @superhx in #2037
• fix(issues2038): fix timestamp to offset not found by @superhx in #2039
• fix(metadata): fix stream endoffset set back by @superhx in #2040
• ci(main): fix start sh by @cuisongliu in #2044
• feat(code): merge apache kafka 3.9.0 6ff51bc by @superhx in #2046
• feat(s3stream): record accumulated value in CounterMetric by @SCNieh in #2047
• feat(auto_balancer): define required metrics in MetricVersion by @SCNieh in #2048
• chore(config): update log4j properties by @Gezi-lzq in #2050
• feat(s3stream): refine network usage statistics by @SCNieh in #2053
• feat(metadata): change KVImage state to TimelineHashMap by @lifepuzzlefun in #2049
• fix(metadata): add getter for KVDelta by @ShadowySpirits in #2056
• fix(test): update base docker file & fix unstable test by @superhx in #2055
• feat(s3stream): add read timeout by @superhx in #2058
• feat(cherry-pick): KAFKA-16927 Handle expanding leader endpoints (#17363) by @superhx in #2060
• feat(auto_balancer): add timestamp param for Samples interface by @SCNieh in #2061
• feat(auto_balancer): use uniformed interface for reconfiguration by @SCNieh in #2062
• fix(scripts): start controller node in daemon mode by @Chillax-0v0 in #2065
• feat(auto_balancer): add missing method by @SCNieh in #2066
• fix(scripts): fix typo by @Chillax-0v0 in #2068
• fix(scripts): fix "advertised.listeners" when deploying cluster by @Chillax-0v0 in #2070
• fix(scripts): add single quotes around the env values by @Chillax-0v0 in #2072
• fix(config): update example config "listeners" for brokers by @Chillax-0v0 in #2074
• fix(test): fix snapshot_test by @superhx in #2076
• perf(kraft): decrease the index interval bytes of KRaft Log from 1MB to 4KB by @Chillax-0v0 in #2078
• perf(s3stream/limiter): decrease the max tokens of network limiters by @Chillax-0v0 in #2077
• feat(metadata): return leader info after partition success failover by @lifepuzzlefun in #2051
• feat(metrics): add metrics on the log append permits by @Chillax-0v0 in #2081
• fix(s3stream): remove unnecessary config check by @SCNieh in #2084
• chore: remove log and add topic check method by @Gezi-lzq in #2086
• fix(network): update metrics defore disconnection by @Chillax-0v0 in #2080
• fix(metadata): try to fix MetadataImage refCount by @lifepuzzlefun in #2089
• docs: Enhance Contributing Guide by @CtrlAltDft in #2085
• fix(core): fix time unit by @SCNieh in #2092
• chore(log): remove checkFailoverSuccess error log by @superhx in #2093
• chore(tools/perf): log GC status by default by @Chillax-0v0 in #2082
• fix(s3stream): fix compaction exit on empty stream metadata list by @SCNieh in #2095
• fix(sasl): fix sasl configs by @Chillax-0v0 in #2097
• refactor: add relationship explaination with kafka in readme by @KaimingWan in #2098
• feat(auto_balancer): refine interface parameters by @SCNieh in #2102
• fix(auto_balancer): fix mistakenly reused reference by @SCNieh in #2104
• feat(merge): merge apache kafka 3.9 398b4c4 by @superhx in #2105
• perf: limit the inflight requests (#2100) by @Chillax-0v0 in #2106
• fix(issue2108): avoid blocking at the end of a compaction iteration when there are un-uploaded data by @SCNieh in #2111
• fix(s3stream): wait force upload complete before return by @superhx in #2113
• fix(compaction): prevent double release on compaction shutdown by @SCNieh in #2116
• fix(s3stream): fix available bandwidth metrics by @SCNieh in #2120
• fix(metrics): fix metrics name for BrokerTopicPartitionMetrics by @SCNieh in #2118
• fix(checkstyle): fix checkstyle by @SCNieh in #2121
• fix(e2e): remove unstable autobalancer tests by @SCNieh in #2123
• revert(s3stream/limiter): increase the max tokens of network limiters by @Chillax-0v0 in #2125
• perf(tools/perf): assuming all partitions have the same offset at the same time by @Chillax-0v0 in #2127
• perf(log): avoid too many checkpoint at the same time by @Chillax-0v0 in #2129
• refactor(tools/perf): retry sending messages in when waiting topics ready by @Chillax-0v0 in #2132
• fix(s3stream/storage): correct if condition on awaitTermination by @Chillax-0v0 in #2137
• feat(tools/perf): run benchmark without consumer by @Chillax-0v0 in #2134
• fix(issue2139): prevent read object info from closed ObjectReader by @SCNieh in #2143
• fix(issue2139): add computeIfAbsent atomic operation to AsyncLRUCache by @SCNieh in #2145
• fix(issue2140): remove override equals and hashCode method for ObjectReader by @SCNieh in #2148
• feat(merge): merge apache kafka 3.9.0 cc53a63 by @superhx in #2150
• feat(version): bump version to 1.3.0-rc0 by @superhx in #2153

New Contributors

• @Pey-crypto made their first contribution in #1879
• @cuisongliu made their first contribution in #2027
• @CtrlAltDft made their first contribution in #2085

Full Changelog: 1.2.0...1.3.0-rc0