fix(clickhouse): stop HTTP Native stream desync on connect driver - #8248
Open
sentry[bot] wants to merge 13 commits into
Open
fix(clickhouse): stop HTTP Native stream desync on connect driver#8248sentry[bot] wants to merge 13 commits into
sentry[bot] wants to merge 13 commits into
Conversation
sentry
Bot
changed the title
phacops
force-pushed
the
seer/fix-clickhouse-connect-dynamic-type
branch
from
August 1, 2026 03:57
7b12720 to
fb7b691
Compare
phacops
changed the title
EAP queries over clickhouse-connect were failing with `Unrecognized ClickHouse type base: <attribute values>` because the HTTP Native decoder lost framing and read row payload as type names. Disable client_protocol_version negotiation (fragile on HTTP), pin lz4 compression, translate StreamFailureError, and retry once after resetting the cached client on desync.
phacops
force-pushed
the
seer/fix-clickhouse-connect-dynamic-type
branch
from
August 1, 2026 03:58
fb7b691 to
9ddaae0
Compare
Replace the hardcoded use_protocol_version=False with clickhouse_connect_use_protocol_version (default false), applied when the HTTP client is created so it can be flipped without a deploy.
Migrate clickhouse_connect_pool_size off runtime state config onto a sentry-option (default 25), matching the rest of the connect driver knobs.
Apply clickhouse_connect_use_protocol_version once at module import, matching invalid_setting_action, instead of re-setting it on every client create.
Emit clickhouse.connect.* for connection errors and stream desync so they sit next to clickhouse.native.* instead of under a mixed clickhouse.connect.<unprefixed> namespace.
Keep metric names unprefixed; clickhouse.connect comes from the wrapper, same as before.
Retry once after close_connections() instead of rebuilding the client, and fold the retry into a short loop.
Each ClickhouseConnectPool talks to one LB origin, so num_pools=1 is enough; the second slot was unused.
Surface desync and stream failures as ClickhouseError immediately; keep metrics, drop connection reset/retry.
On transport errors, stream failures, and Native desync, clear the HTTP connection pool before surfacing ClickhouseError so the next query does not reuse a poisoned socket. Still no query retry.
Expose clickhouse_connect_connect_timeout and clickhouse_connect_send_receive_timeout (0 keeps constructor/profile values) for HTTP client creation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
EAP queries on the clickhouse-connect (HTTP) driver fail intermittently with:
This is a Native binary framing desync: the decoder reads row payload (attribute strings) as column type names. HTTP-only, unrelated to Dynamic casting.
Changes
client_protocol_versionby default via sentry-optionclickhouse_connect_use_protocol_version(plain Native framing; safer when hops strip the setting)compress="lz4"andnum_pools=1(single LB origin)StreamFailureErrorand desync intoClickhouseErrorclickhouse.connect.connection_error,stream_failure,stream_desyncclickhouse_connect_pool_sizeclickhouse_connect_connect_timeout(0= constructor default)clickhouse_connect_send_receive_timeout(0= profile/constructor value)Legal Boilerplate
Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.
Fixes SNUBA-BMW