23.3.19 Altinity Stable

Changes compared to upstream's v23.3.19.32-lts

Bug fixes

• Fix key condition on duplicate primary keys (ClickHouse#48838 by @amosbird via #275 )
• Fix excessive memory usage for FINAL (ClickHouse#50429 by @ azat via #307)
• Fixed DNS-related crash (ClickHouse#52634 by @arthurpassos via #295)
• Fix parquet array inconsistent offsets (ClickHouse#54370 by @arthurpassos via #303)
• Fixed rare bug when we produced invalid read ranges for queries with FINAL. Resulted in Cannot read out of marks range exception (ClickHouse#54934 by @nickitat via #313)
• Fixed reading from sparse columns after changing setting ratio_of_defaults_for_sparse_serialization back to 1.0 and restarting the server. (ClickHouse#49660 by @CurtizJ via #314)
• Fix incorrect free space accounting for least_used JBOD policy (ClickHouse#56030 by @azat via #365)
• Fixes race condition, prevents application crash (ClickHouse#52623 by @hanfei1991 via #366)
• Fixed leaking loggers in some cases, like creating and dropping tables (ClickHouse#58831, ClickHouse#59273 by @kitaisreal via #356)
• Fix deadlock in LDAP assigned role update for non-existing ClickHouse roles (ClickHouse#55119 by @jmaicher, ClickHouse#56544 by @jmaicher via #364)

Performance improvement

• Do not build sets for skip indexes if they are not used (ClickHouse#48299 by @CurtizJ via #276)

Improvement

• Disabled unsafe "compile_expressions" setting by default (ClickHouse#51368 by @alexey-milovidov via #306)
• Row policy for database (ClickHouse#47640 by @ilejn via #300)

Build/Testing/Packaging Improvement

• Bring back DNS tests (ClickHouse#53286 - @arthurpassos via #296)
• Made builds and tests possible in Altinity's infrastructure
• Support of Aarch64 builds (#311)
  • building Aarch64 binaries
  • multi-arch docker containers (Amd64, Aarch64)
  • running stateles, stateful tests against Aarch64
  • running clickhouse-regression tests against Aarch64

upstream ClickHouse changes between v23.3.19.32-lts (c4d4ca8ec02) and v23.3.13.6-lts (25635e27551)

Improvement

• Fix transfer query to MySQL compatible query. (ClickHouse#56456 by flynn via ClickHouse#58489).
• There was a potential vulnerability in previous ClickHouse versions: if a user has connected and unsuccessfully tried to authenticate with the "interserver secret" method, the server didn't terminate the connection immediately but continued to receive and ignore the leftover packets from the client. While these packets are ignored, they are still parsed, and if they use a compression method with another known vulnerability, it will lead to exploitation of it without authentication. This issue was found with [ClickHouse Bug Bounty Program] by https://twitter.com/malacupa. (ClickHouse#56794 by Alexey Milovidov via ClickHouse#56928).
• Handle sigabrt case when getting PostgreSQl table structure with empty array. (ClickHouse#57618 by Mikhail Kot via ClickHouse#57653).

Build/Testing/Packaging Improvement

• Fix strange additional QEMU logging after ClickHouse#47151, see https://s3.amazonaws.com/clickhouse-test-reports/50078/a4743996ee4f3583884d07bcd6501df0cfdaa346/stateless_tests__release__databasereplicated__[3_4].html. (ClickHouse#50442 by Mikhail f. Shiryaev via ClickHouse#54312).
• If the database is already initialized, it doesn't need to be initialized again upon subsequent launches. This can potentially fix the issue of infinite container restarts when the database fails to load within 1000 attempts (relevant for very large databases and multi-node setups). (ClickHouse#50724 by Alexander Nikolaev via ClickHouse#55671).
• Resource with source code including submodules is built in Darwin special build task. It may be used to build ClickHouse without checkouting submodules. (ClickHouse#51435 by Ilya Yatsishin via ClickHouse#55391).
• Enrich changed_images.json with the latest tag from master for images that are not changed in the pull request. (ClickHouse#54369 by János Benjamin Antal via ClickHouse#54703).
• We build and upload them for every push, which isn't worth it. (ClickHouse#54675 by Mikhail f. Shiryaev via ClickHouse#54679).
• Fix integration check python script to use gh api url - Add Readme for CI tests. (ClickHouse#55716 by Max K. via ClickHouse#55734).
• Check sha512 for tgz; use a proper repository for keeper; write only filenames to TGZ.sha512 files for tarball packages. Prerequisite for ClickHouse#31473. (ClickHouse#55717 by Mikhail f. Shiryaev via ClickHouse#55829).
• Do not fetch changed submodules in the builder container. (ClickHouse#56689 by Mikhail f. Shiryaev via ClickHouse#56731).
• There was an attempt to have the proper listing in ClickHouse#44311, but the fix itself was in the wrong place, so it's still broken. See an example. (ClickHouse#56989 by Mikhail f. Shiryaev via ClickHouse#57019).
• Fix issue caught in docker-library/official-images#15846. (ClickHouse#57571 by Mikhail f. Shiryaev via ClickHouse#57580).
• Pin rust version to fix GLIBC compatibility check. (ClickHouse#55788 by Mikhail f. Shiryaev).

Bug Fix (user-visible misbehavior in an official stable release)

• Fix parameterized view with cte and multiple usage (ClickHouse#52328 by SmitaRKulkarni).
• Check recursion depth in OptimizedRegularExpression (ClickHouse#52451 by Alexey Milovidov).
• Fix lightweight delete after drop of projection (ClickHouse#52517 by Anton Popov).
• Fix sorting of sparse columns with large limit (ClickHouse#52827 by Anton Popov).
• Fix adding sub-second intervals to DateTime (ClickHouse#53309 by Michael Kolupaev).
• Fix: allow IPv6 for bloom filter (ClickHouse#54200 by Yakov Olkhovskiy).
• Check for overflow before addition in analysisOfVariance function (ClickHouse#54385 by Antonio Andelic).
• Fix aggregate projections with normalized states (ClickHouse#54480 by Amos Bird).
• Fix possible parsing error in WithNames formats with disabled input_format_with_names_use_header (ClickHouse#54513 by Kruglov Pavel).
• Fix: avoid using regex match, possibly containing alternation, as a key condition. (ClickHouse#54696 by Yakov Olkhovskiy).
• Fix "Invalid number of rows in Chunk" in MaterializedPostgreSQL (ClickHouse#54844 by Kseniia Sumarokova).
• Prevent attaching parts from tables with different projections or indices (ClickHouse#55062 by János Benjamin Antal).
• Fix bug with inability to drop detached partition in replicated merge tree on top of S3 without zero copy (ClickHouse#55309 by alesapin).
• Fix trash optimization (up to a certain extent) (ClickHouse#55353 by Alexey Milovidov).
• Fix parsing of arrays in cast operator (ClickHouse#55417 by Anton Popov).
• Fix crash in QueryNormalizer with cyclic aliases (ClickHouse#55602 by vdimir).
• Fix window functions in case of sparse columns. (ClickHouse#55895 by János Benjamin Antal).
• Fix segfault during Kerberos initialization (ClickHouse#56401 by Nikolay Degterinsky).
• Fix buffer overflow in T64 (ClickHouse#56434 by Alexey Milovidov).
• Fix crash in FPC codec (ClickHouse#56795 by Alexey Milovidov).
• Prevent incompatible ALTER of projection columns (ClickHouse#56948 by Amos Bird).
• Fix buffer overflow in Gorilla codec (ClickHouse#57107 by Nikolay Degterinsky).
• Close interserver connection on any exception before authentication (ClickHouse#57142 by Antonio Andelic).
• Fix segfault after ALTER UPDATE with Nullable MATERIALIZED column (ClickHouse#57147 by Nikolay Degterinsky).
• Fix incorrect JOIN plan optimization with partially materialized normal projection (ClickHouse#57196 by Amos Bird).
• MergeTree mutations reuse source part index granularity (ClickHouse#57352 by Maksim Kita).
• Fix invalid memory access in BLAKE3 (Rust) (ClickHouse#57876 by Raúl Marín).
• Normalize function names in CREATE INDEX (ClickHouse#57906 by Alexander Tokmakov).
• Fix invalid preprocessing on Keeper (ClickHouse#58069 by Antonio Andelic).
• Fix Integer overflow in Poco::UTF32Encoding (ClickHouse#58073 by Andrey Fedotov).
• Remove parallel parsing for JSONCompactEachRow (ClickHouse#58181 by Alexey Milovidov).
• Fix incorrect createColumn call on join clause (ClickHouse#48998 by Yi Sheng).

NOT FOR CHANGELOG / INSIGNIFICANT

• Test libunwind changes. (ClickHouse#51436 by Nikolai Kochetov).
• Refactor CI_CONFIG (ClickHouse#52948 by Mikhail f. Shiryaev).
• Use pathlib.Path in S3Helper, rewrite build reports, improve small things (ClickHouse#54010 by Mikhail f. Shiryaev).
• Fixed parameterized_view test after backporting a fix 23.3 (ClickHouse#54401 by SmitaRKulkarni).
• Update automated commit status comment (ClickHouse#54441 by vdimir).
• S3 artifacts (ClickHouse#54504 by Mikhail f. Shiryaev).
• Fix CI skip build and skip tests checks (ClickHouse#54532 by SmitaRKulkarni).
• Update WebObjectStorage.cpp (ClickHouse#54695 by Kseniia Sumarokova).
• Do not set PR status label (ClickHouse#54799 by vdimir).
• Use --filter to reduce checkout time (ClickHouse#54857 by Mikhail f. Shiryaev).
• Get rid of the most of os.path stuff (ClickHouse#55028 by Mikhail f. Shiryaev).
• check if block is empty after async insert retries (ClickHouse#55143 by Han Fei).
• MaterializedPostgreSQL: remove back check (ClickHouse#55297 by Kseniia Sumarokova).
• Remove existing moving/ dir if allow_remove_stale_moving_parts is off (ClickHouse#55480 by Mikhail Kot).
• Improve enrich image (ClickHouse#55793 by Mikhail f. Shiryaev).
• One final leftover in diff_urls of PRInfo (ClickHouse#55874 by Mikhail f. Shiryaev).
• Rewrite jobs to use callable workflow (ClickHouse#56385 by Mikhail f. Shiryaev).
• Continue rewriting workflows to reusable tests (ClickHouse#56501 by Mikhail f. Shiryaev).
• Fix pygithub (ClickHouse#56778 by Mikhail f. Shiryaev).
• Better exception messages (ClickHouse#56854 by Antonio Andelic).
• Avoid dependencies with no fixed versions (ClickHouse#56914 by Alexey Milovidov).
• Tiny improvement security (ClickHouse#57171 by Mikhail f. Shiryaev).
• Pin alpine version of integration tests helper container (ClickHouse#57669 by Mikhail f. Shiryaev).
• Fix docker image for integration tests (fixes CI) (ClickHouse#57952 by Azat Khuzhin).