Using Lightweight Deletion

[kannan-k-s]

im just a newbie for clickhouse and looking way for syncing mysql and clickhouse.

The sink connector tool looks good, but i think the tool also has the ability to address the duplication problem in ReplacingMergeTree family tables.

Instead of waiting for the merge has to be done in background, why not we have a in sync light weight deletion in updates & deletion cases??

[aadant]

RMT is the best option for OLTP workloads to CH
Lightweight deletes come with limitations ClickHouse/ClickHouse#39870

[kannan-k-s]

We expect no duplication at any time, so RMT doesn't look like an only option (up to what i tried, may be im missing something) so i was looking to something like LWD.

any other ideas for better deduplication will be helpful.

Thanks in advance.

[aadant]

If you use kafka, there is no way to guarantee unique delivery because partition can be rebalanced.

if CH goes down, the sink connector will resend what was not acked and may or may not be inserted. CH does not have unique constraints.

RMT has the same perf as MT if you do not use final or if the data is merged . CH optimized a lot of final workloads.