fix(clickhouse): suppress ON CLUSTER for object DDL in a Replicated database - #6003
Draft
mday-io wants to merge 1 commit into
Draft
fix(clickhouse): suppress ON CLUSTER for object DDL in a Replicated database#6003mday-io wants to merge 1 commit into
mday-io wants to merge 1 commit into
Conversation
…atabase
Inside a `Replicated` database Keeper already propagates object DDL, so adding
`ON CLUSTER` asks for a second, redundant fan-out and ClickHouse rejects the
statement outright:
Code: 80. DB::Exception: It's not initial query.
ON CLUSTER is not allowed for Replicated database. (INCORRECT_QUERY)
The ClickHouse adapter had no way to avoid this: `_on_cluster_sql()` took no
argument and keyed only on `engine_run_mode.is_cluster`. The practical effect is
total -- with a cluster configured, no table or view can be created in a
`Replicated` database at all. Reproduced end to end against a 1x3 cluster on
26.6.3.62: `create_table` into such a database fails with the error above before
this change and succeeds after it, propagating to all three replicas under one
shared UUID.
This cannot be a connection-level flag. One connection can hold both an Atomic
and a `Replicated` database, and objects in the Atomic one still need
`ON CLUSTER`, so the decision is made per target database by
`_should_use_on_cluster()` and threaded through all 14 emission sites.
The policy fails open throughout. An unknown target, an unresolvable database,
or any introspection failure keeps the previous behaviour, and a single cached
probe short-circuits the whole path when the server hosts no `Replicated`
database at all -- so this is a no-op, and costs no extra round-trips, for any
deployment that does not have one.
Two carve-outs are deliberate. `CREATE DATABASE` is how a `Replicated` database
comes into existence, so there is no engine to consult, and a database's own DDL
log cannot propagate its own removal; both stay cluster-wide unconditionally.
`alter_table` decides per expression rather than per call, because one call can
carry alters against both kinds of database, and `RENAME` / `EXCHANGE` across a
Replicated and a non-Replicated database are refused rather than rendered
silently half-correct.
`_build_table_properties_exp` and `_build_view_properties_exp` receive no
reference to the object being created, so the base class now passes the target
table down. The parameter is additive and every override across the nine adapter
files already accepts `**kwargs`, so no other adapter changes; Athena already
declares and receives the same `table` parameter via its own override.
This is a correctness fix independent of managed models: any SQLMesh user
running against a `Replicated` database is affected today for ordinary tables
and views.
mday-io
force-pushed
the
mday/clickhouse-replicated-on-cluster
branch
from
August 31, 2026 15:07
f68f12f to
b6c3e72
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Inside a
Replicateddatabase, Keeper already propagates object DDL to every replica. AddingON CLUSTERasks for a second, redundant fan-out, and ClickHouse does not silently double-apply it — it rejects the statement outright:The ClickHouse adapter had no way to avoid this.
_on_cluster_sql()took no argument and keyed only onengine_run_mode.is_cluster, so with a cluster configured SQLMesh cannot create any table or view in aReplicateddatabase. This is a correctness bug independent of managed models: it affects ordinary tables and views today.Reproduced end to end against a 1x3 cluster on ClickHouse 26.6.3.62 —
create_tableinto aReplicateddatabase fails with the error above before this change and succeeds after it, propagating to all three replicas under one shared UUID.Approach
This cannot be a connection-level flag. One connection can hold both an
Atomicand aReplicateddatabase, and objects in theAtomicone still needON CLUSTER. So the decision is made per target database by a new_should_use_on_cluster(), threaded through everyON CLUSTERemission site.The policy fails open throughout: an unknown target, an unresolvable database, or any introspection failure keeps the previous behaviour.
Two carve-outs are deliberate:
CREATE DATABASEis how aReplicateddatabase comes into existence, so there is no engine to consult, and a database's own DDL log cannot propagate its own removal. Both stay cluster-wide unconditionally.alter_tabledecides per expression rather than per call, since one call can carry alters against both kinds of database.RENAME/EXCHANGEspanning aReplicatedand a non-Replicateddatabase are refused rather than rendered silently half-correct.What this costs deployments that don't have a Replicated database
Nothing — and that's about round-trips, not only rendered SQL.
The naive form of this predicate would introspect on essentially every DDL statement. A
_has_replicated_databaseshort-circuit avoids that: one cached probe per connection (SELECT count() FROM system.databases WHERE engine LIKE 'Replicated%'). When the server hosts noReplicateddatabase — the overwhelmingly common case — the predicate returns immediately without resolving a target or querying per database. The cache is invalidated alongside the engine cache oncreate_schemaand database drops.Base-class change
_build_table_properties_expand_build_view_properties_expreceive no reference to the object being created, so the base class now passes the target table down. The parameter is additive and every override across the nine adapter files already accepts**kwargs, so no other adapter changes are needed; Athena already declares and receives the sametableparameter via its own override.Scope note
This fix is necessary but not sufficient for coordinated refreshable materialized views on a multi-shard
Replicatedtopology — that additionally requires a server-macro deployment contract, which is not adapter code. The two are independent; please review this on its own terms as a DDL correctness fix.Testing
tests/core/engine_adapter/test_clickhouse.py: 46 passed (34 existing + 12 new). New tests cover the predicate, per-site rendering againstAtomicvsReplicatedtargets, both carve-outs, the per-expressionalter_tablecase, the cross-engine rename/exchange refusals, cache behaviour, and the no-op short-circuit.tests/core/engine_adapter/andtests/core/test_snapshot_evaluator.py: 783 passed. The 2test_snowflake.pyfailures reproduce on a clean checkout of this base commit and are unrelated;test_spark.pyand the live-engine integration suites were not collected in this environment.ruff format/ruff checkclean;mypyreports 75 errors both before and after, i.e. zero new.