Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support for timescaledb(postgres variant) - Added in Debezium 2.4.0 #886

Open
subkanthi opened this issue Oct 28, 2024 · 2 comments
Open
Milestone

Comments

@subkanthi
Copy link
Collaborator

A request from a user in altinitydb

https://debezium.io/blog/2024/01/11/Debezium-and-TimescaleDB/

@salisbury-espinosa
Copy link
Contributor

but it seems like everything is working as is...
timescaledb - it's just an extension - so it all work

TimescaleDB is an extension for PostgreSQL that enables time-series workloads, increasing ingest, query, storage and analytics performance.

@subkanthi subkanthi modified the milestones: 2.5.0, 2.6.0 Nov 6, 2024
@salisbury-espinosa
Copy link
Contributor

salisbury-espinosa commented Nov 7, 2024

there is a tricky setup of the data transfer from the postgresql itself to Kafka, because in timescale there is no native partitioning - but chunks - physical tables and there the timescale itself and the publication during logical replication must be cunningly, but very simply configured.
and this only concerns the setup of the postgres link to Kafka via Debezium.
and does not concern the sinker (simply because the sinker works with Kafka in Debezium format and it does not concern itself with how the data gets to this very Kafka).
and these are 5 lines of settings and do not affect the code of "clickhouse-sink-connector" in any way

by the way, the sinker works only in the original Debezium format - for example, you can't use such settings as:

transforms=unwrap,...
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState

because the sinker expects a specific data format:

https://github.com/Altinity/clickhouse-sink-connector/blob/develop/sink-connector/src/main/java/com/altinity/clickhouse/sink/connector/model/SinkRecordColumns.java

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants