Skip to content

fix(clickhouse): suppress ON CLUSTER for object DDL in a Replicated database - #6003

Draft
mday-io wants to merge 1 commit into
SQLMesh:mainfrom
mday-io:mday/clickhouse-replicated-on-cluster
Draft

fix(clickhouse): suppress ON CLUSTER for object DDL in a Replicated database#6003
mday-io wants to merge 1 commit into
SQLMesh:mainfrom
mday-io:mday/clickhouse-replicated-on-cluster

Conversation

@mday-io

@mday-io mday-io commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Problem

Inside a Replicated database, Keeper already propagates object DDL to every replica. Adding ON CLUSTER asks for a second, redundant fan-out, and ClickHouse does not silently double-apply it — it rejects the statement outright:

Code: 80. DB::Exception: It's not initial query.
ON CLUSTER is not allowed for Replicated database. (INCORRECT_QUERY)

The ClickHouse adapter had no way to avoid this. _on_cluster_sql() took no argument and keyed only on engine_run_mode.is_cluster, so with a cluster configured SQLMesh cannot create any table or view in a Replicated database. This is a correctness bug independent of managed models: it affects ordinary tables and views today.

Reproduced end to end against a 1x3 cluster on ClickHouse 26.6.3.62 — create_table into a Replicated database fails with the error above before this change and succeeds after it, propagating to all three replicas under one shared UUID.

Approach

This cannot be a connection-level flag. One connection can hold both an Atomic and a Replicated database, and objects in the Atomic one still need ON CLUSTER. So the decision is made per target database by a new _should_use_on_cluster(), threaded through every ON CLUSTER emission site.

The policy fails open throughout: an unknown target, an unresolvable database, or any introspection failure keeps the previous behaviour.

Two carve-outs are deliberate:

  • CREATE DATABASE is how a Replicated database comes into existence, so there is no engine to consult, and a database's own DDL log cannot propagate its own removal. Both stay cluster-wide unconditionally.
  • alter_table decides per expression rather than per call, since one call can carry alters against both kinds of database. RENAME / EXCHANGE spanning a Replicated and a non-Replicated database are refused rather than rendered silently half-correct.

What this costs deployments that don't have a Replicated database

Nothing — and that's about round-trips, not only rendered SQL.

The naive form of this predicate would introspect on essentially every DDL statement. A _has_replicated_database short-circuit avoids that: one cached probe per connection (SELECT count() FROM system.databases WHERE engine LIKE 'Replicated%'). When the server hosts no Replicated database — the overwhelmingly common case — the predicate returns immediately without resolving a target or querying per database. The cache is invalidated alongside the engine cache on create_schema and database drops.

Base-class change

_build_table_properties_exp and _build_view_properties_exp receive no reference to the object being created, so the base class now passes the target table down. The parameter is additive and every override across the nine adapter files already accepts **kwargs, so no other adapter changes are needed; Athena already declares and receives the same table parameter via its own override.

Scope note

This fix is necessary but not sufficient for coordinated refreshable materialized views on a multi-shard Replicated topology — that additionally requires a server-macro deployment contract, which is not adapter code. The two are independent; please review this on its own terms as a DDL correctness fix.

Testing

  • tests/core/engine_adapter/test_clickhouse.py: 46 passed (34 existing + 12 new). New tests cover the predicate, per-site rendering against Atomic vs Replicated targets, both carve-outs, the per-expression alter_table case, the cross-engine rename/exchange refusals, cache behaviour, and the no-op short-circuit.
  • Regression gate across tests/core/engine_adapter/ and tests/core/test_snapshot_evaluator.py: 783 passed. The 2 test_snowflake.py failures reproduce on a clean checkout of this base commit and are unrelated; test_spark.py and the live-engine integration suites were not collected in this environment.
  • ruff format / ruff check clean; mypy reports 75 errors both before and after, i.e. zero new.

…atabase

Inside a `Replicated` database Keeper already propagates object DDL, so adding
`ON CLUSTER` asks for a second, redundant fan-out and ClickHouse rejects the
statement outright:

    Code: 80. DB::Exception: It's not initial query.
    ON CLUSTER is not allowed for Replicated database. (INCORRECT_QUERY)

The ClickHouse adapter had no way to avoid this: `_on_cluster_sql()` took no
argument and keyed only on `engine_run_mode.is_cluster`. The practical effect is
total -- with a cluster configured, no table or view can be created in a
`Replicated` database at all. Reproduced end to end against a 1x3 cluster on
26.6.3.62: `create_table` into such a database fails with the error above before
this change and succeeds after it, propagating to all three replicas under one
shared UUID.

This cannot be a connection-level flag. One connection can hold both an Atomic
and a `Replicated` database, and objects in the Atomic one still need
`ON CLUSTER`, so the decision is made per target database by
`_should_use_on_cluster()` and threaded through all 14 emission sites.

The policy fails open throughout. An unknown target, an unresolvable database,
or any introspection failure keeps the previous behaviour, and a single cached
probe short-circuits the whole path when the server hosts no `Replicated`
database at all -- so this is a no-op, and costs no extra round-trips, for any
deployment that does not have one.

Two carve-outs are deliberate. `CREATE DATABASE` is how a `Replicated` database
comes into existence, so there is no engine to consult, and a database's own DDL
log cannot propagate its own removal; both stay cluster-wide unconditionally.
`alter_table` decides per expression rather than per call, because one call can
carry alters against both kinds of database, and `RENAME` / `EXCHANGE` across a
Replicated and a non-Replicated database are refused rather than rendered
silently half-correct.

`_build_table_properties_exp` and `_build_view_properties_exp` receive no
reference to the object being created, so the base class now passes the target
table down. The parameter is additive and every override across the nine adapter
files already accepts `**kwargs`, so no other adapter changes; Athena already
declares and receives the same `table` parameter via its own override.

This is a correctness fix independent of managed models: any SQLMesh user
running against a `Replicated` database is affected today for ordinary tables
and views.
@mday-io
mday-io force-pushed the mday/clickhouse-replicated-on-cluster branch from f68f12f to b6c3e72 Compare August 31, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant