Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Custom replication slot name #540

Closed
edgarrmondragon opened this issue Nov 12, 2024 · 1 comment · Fixed by #543
Closed

feat: Custom replication slot name #540

edgarrmondragon opened this issue Nov 12, 2024 · 1 comment · Fixed by #543
Assignees
@RobyBen
Labels
enhancement New feature or request good first issue Good for newcomers python Pull requests that update Python code

Comments

@edgarrmondragon
Copy link
Member

Hi everyone. I'm using tap-postgres with LOG_BASED replication method. I've successfully setup the logical replication and now able to capture insert, update, and deletes on my database.

I'm running into an issue that my replication slot cannot be used by parallel extractions from the database. Is there a way to set a different replication slot name for each extractor other than tappostgres?

This should be as easy as adding new setting to tap_postgres/tap.py and passing it to

tap-postgres/tap_postgres/client.py

Lines 343 to 353 in e6dd19c

logical_replication_cursor.start_replication(
slot_name="tappostgres",
decode=True,
start_lsn=start_lsn,
status_interval=status_interval,
options={
"format-version": 2,
"include-transaction": False,
"add-tables": self.fully_qualified_name,
},
)

Slack Thread: https://meltano.slack.com/archives/C069CQNHDNF/p1731396392068079
@edgarrmondragon edgarrmondragon added enhancement New feature or request good first issue Good for newcomers python Pull requests that update Python code labels Nov 12, 2024
@RobyBen RobyBen mentioned this issue Nov 23, 2024
Merged
@RobyBen
Copy link
Contributor

RobyBen commented Nov 23, 2024

I made pull request. Check pls

@edgarrmondragon edgarrmondragon moved this from Todo to In Progress in MeltanoLabs Overview Nov 24, 2024
@edgarrmondragon edgarrmondragon linked a pull request Nov 24, 2024 that will close this issue
feat: Custom replication slot name #543
Merged
github-merge-queue bot pushed a commit that referenced this issue Dec 3, 2024
@RobyBen @edgarrmondragon
feat: Custom replication slot name (#543)
25bce26 
Issue #540
**client.py**
Introduced support for specifying replication_slot_name in the
replication logic
Added replication=database to the connection string
Updated application_name

**tap.py**
Validation for replication_slot_name: <=60 characters, and do not start
with pg_.
Added replication_slot_name as a configurable parameter in
config_jsonschema, with a default value of "tappostgres".
Passed replication_slot_name to the PostgresConnector.
Added configuration examples

**Added test_slot_name.py to the tests folder**

---------

Co-authored-by: Edgar Ramírez Mondragón <[email protected]>
Co-authored-by: Edgar Ramírez-Mondragón <[email protected]>
@github-project-automation github-project-automation bot moved this from In Progress to Done in MeltanoLabs Overview Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@RobyBen RobyBen

Labels
enhancement New feature or request good first issue Good for newcomers python Pull requests that update Python code
Projects
MeltanoLabs Overview
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: Custom replication slot name RobyBen/tap-postgres
2 participants
@edgarrmondragon @RobyBen