feat: Taps can now opt-in to emitting `ACTIVATE_VERSION` messages for streams with `FULL_TABLE` replication #2686

edgarrmondragon · 2024-09-24T16:58:39Z

Integration test plan: E2E snaphots accomplish this
Spec compliance tests. Do snapshots cover this?
Docs: docs: Document ACTIVATE_VERSION messages and capability #3079

Summary by Sourcery

Emit ACTIVATE_VERSION messages and attach version metadata for FULL_TABLE replication streams; adjust replication key behavior, batch messages, and sample mapper logic; and bolster snapshot-based tests for tap→mapper→target workflows.

New Features:

Enable taps to emit ACTIVATE_VERSION messages and include version metadata for streams using FULL_TABLE replication

Enhancements:

Use tap initialization time for replication key signposts in timestamp-based replication
Introduce a version field in RecordMessage and SDKBatchMessage to track table versions
Skip RemoveRecordTransform when mapping ActivateVersionMessage in sample mapper
Update codespell config to ignore JSONL snapshot files

Tests:

Add snapshot-driven end-to-end tests for tap→mapper→target flows with time-machine for deterministic timestamps
Expand and update sample and core test suites with new fixtures and snapshot files

Summary by Sourcery

Enable taps to optionally emit ACTIVATE_VERSION messages and include version metadata for FULL_TABLE replication streams, extend core stream and message classes to support version tracking, update sample mapper and capabilities, and add snapshot-driven tests for tap→mapper→target workflows.

New Features:

Add opt-in support for emitting ACTIVATE_VERSION messages with version metadata for FULL_TABLE replication streams.

Enhancements:

Add a version field to RecordMessage and SDKBatchMessage to track table version.
Use tap initialization timestamp as the replication key signpost for timestamp-based replication.
Introduce emit_activate_version_messages config property and update tap capabilities to include ACTIVATE_VERSION.
Skip RemoveRecordTransform when mapping ActivateVersionMessage in the sample mapper.
Update codespell configuration to ignore JSONL snapshot files.

Tests:

Add snapshot-driven end-to-end tests for CSV and SQLite targets covering activate_version, hard delete, and missing properties scenarios.
Refactor tests to use pytest snapshots, caplog, time_machine fixture, and deterministic table names using request.node.name.

Chores:

Adjust existing test setup to use pytest snapshot plugin and remove UUID-based table naming.

codspeed-hq · 2024-09-24T17:01:32Z

CodSpeed Performance Report

Merging #2686 will not alter performance

_{Comparing edgarrmondragon/feat/activate_version (1acde43) with main (9497d9b)}

Summary

✅ 8 untouched benchmarks

codecov · 2024-09-24T17:02:04Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.13%. Comparing base (9497d9b) to head (1acde43).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2686      +/-   ##
==========================================
+ Coverage   92.01%   92.13%   +0.11%     
==========================================
  Files          62       62              
  Lines        5298     5312      +14     
  Branches      682      683       +1     
==========================================
+ Hits         4875     4894      +19     
+ Misses        297      293       -4     
+ Partials      126      125       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

… streams with `FULL_TABLE` replication

edgarrmondragon · 2025-06-03T03:27:21Z

@sourcery-ai review

sourcery-ai · 2025-06-03T03:32:04Z

Reviewer's Guide

This PR adds an optional ACTIVATE_VERSION message emission flow for full‐table taps with version metadata, refines replication key timing, updates sample mapper logic, and bolsters snapshot‐driven end‐to‐end tests across CSV and SQLite targets.

File-Level Changes

Change	Details	Files
Configurable emission of `ACTIVATE_VERSION` messages in taps	Introduce `emit_activate_version_messages` config property and capability flag Expose `emit_activate_version_messages` property on streams Set `_stream_version` from tap initialization time and write activate messages in `sync` Add `_write_activate_version_message` helper in core stream logic	`singer_sdk/streams/core.py` `singer_sdk/helpers/capabilities.py` `singer_sdk/tap_base.py`
Attach version metadata to record and batch messages	Include `version` on `RecordMessage` output Add `version` field to `SDKBatchMessage` for full table batches	`singer_sdk/streams/core.py` `singer_sdk/helpers/_batch.py`
Use tap initialization time for replication key signpost	Return `initialized_at` timestamp for timestamp‐based replication key signpost	`singer_sdk/streams/core.py`
Skip removal transform when mapping activate version messages	Bypass `RemoveRecordTransform` in `map_activate_version_message`	`samples/sample_mapper/mapper.py`
Enhance snapshot‐driven end‐to‐end tests for CSV and SQLite	Parametrize CSV mapper/target tests to cover both emit and no‐emit flows Add `pytest.mark.snapshot`, caplog assertions, and use `request.node.name` for table names in SQLite tests Update snapshot directories and files for new activate version and log outputs	`tests/samples/test_target_csv.py` `tests/samples/test_target_sqlite.py` `tests/core/test_parent_child.py` `tests/samples/snapshots/**`

Assessment against linked issues

Issue	Objective	Addressed	Explanation

Possibly linked issues

docs: Document ACTIVATE_VERSION messages and capability #3079: The PR adds the functionality for taps to emit ACTIVATE_VERSION messages, which is what the issue is documenting.

Tips and commands

Interacting with Sourcery

Trigger a new review: Comment @sourcery-ai review on the pull request.
Continue discussions: Reply directly to Sourcery's review comments.
Generate a GitHub issue from a review comment: Ask Sourcery to create an
issue from a review comment by replying to it. You can also reply to a
review comment with @sourcery-ai issue to create an issue from it.
Generate a pull request title: Write @sourcery-ai anywhere in the pull
request title to generate a title at any time. You can also comment
@sourcery-ai title on the pull request to (re-)generate the title at any time.
Generate a pull request summary: Write @sourcery-ai summary anywhere in
the pull request body to generate a PR summary at any time exactly where you
want it. You can also comment @sourcery-ai summary on the pull request to
(re-)generate the summary at any time.
Generate reviewer's guide: Comment @sourcery-ai guide on the pull
request to (re-)generate the reviewer's guide at any time.
Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
pull request to resolve all Sourcery comments. Useful if you've already
addressed all the comments and don't want to see them anymore.
Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
request to dismiss all existing Sourcery reviews. Especially useful if you
want to start fresh with a new review - don't forget to comment
@sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

Enable or disable review features such as the Sourcery-generated pull request
summary, the reviewer's guide, and others.
Change the review language.
Add, remove or edit custom review instructions.
Adjust other review settings.

Getting Help

Contact our support team for questions or feedback.
Visit our documentation for detailed guides and information.
Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

sourcery-ai

Hey @edgarrmondragon - I've reviewed your changes and they look great!

Here's what I looked at during the review

🟡 General issues: 1 issue found
🟢 Security: all looks good
🟡 Testing: 2 issues found
🟢 Documentation: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

singer_sdk/streams/core.py

sourcery-ai · 2025-06-03T03:32:05Z

tests/samples/test_target_csv.py

-def test_countries_to_csv_mapped(csv_config: dict):
+@time_machine.travel(DATETIME, tick=False)
+@pytest.mark.snapshot
+def test_countries_to_csv_mapped(


suggestion (testing): Consider testing behavior for non-FULL_TABLE replication streams.

Add a test with a non-FULL_TABLE replication method (e.g., INCREMENTAL) to confirm that ACTIVATE_VERSION messages are not emitted and RECORD messages lack the new stream version field.

tests/samples/test_target_csv.py

sourcery-ai

Hey @edgarrmondragon - I've reviewed your changes and they look great!

Here's what I looked at during the review

🟡 General issues: 2 issues found
🟢 Security: all looks good
🟢 Testing: all looks good
🟢 Documentation: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

singer_sdk/streams/core.py

edgarrmondragon self-assigned this Oct 15, 2024

edgarrmondragon force-pushed the edgarrmondragon/feat/activate_version branch from fe4d249 to 722567c Compare May 21, 2025 20:37

feat: Taps can now opt-in to emitting ACTIVATE_VERSION messages for…

87cff00

… streams with `FULL_TABLE` replication

edgarrmondragon force-pushed the edgarrmondragon/feat/activate_version branch from 722567c to 87cff00 Compare May 21, 2025 20:40

edgarrmondragon added 9 commits May 21, 2025 14:44

chore: Update snapshots

6594743

Merge branch 'main' into edgarrmondragon/feat/activate_version

8beb639

Convert self._initialized_at to seconds

4610ea3

Update snapshots

5db43dc

Check missing columns

b294e88

Emit message only is stream is selected

a8c6773

Freeze initialization time

f6e4182

Make snapshots for E2E tap > mapper > target test

17b4c2a

Fix target input

228d09b

sourcery-ai bot reviewed Jun 3, 2025

View reviewed changes

Use consistent stream version

0c8729e

edgarrmondragon modified the milestones: v1.0 Release, v0.47 Jun 3, 2025

Merge branch 'main' into edgarrmondragon/feat/activate_version

18de4a8

edgarrmondragon force-pushed the edgarrmondragon/feat/activate_version branch from 3e0ceb3 to 18de4a8 Compare June 3, 2025 23:15

edgarrmondragon added 4 commits June 6, 2025 09:32

Merge branch 'main' into edgarrmondragon/feat/activate_version

cc4399a

Make it opt-in

505b94c

Update tests

6c709c1

Parametrize tap > mapper > target test

1acde43

edgarrmondragon mentioned this pull request Jun 6, 2025

docs: Document ACTIVATE_VERSION messages and capability #3079

Open

edgarrmondragon marked this pull request as ready for review June 6, 2025 17:53

edgarrmondragon added this pull request to the merge queue Jun 6, 2025

Merged via the queue into main with commit 0373c6e Jun 6, 2025
36 checks passed

edgarrmondragon deleted the edgarrmondragon/feat/activate_version branch June 6, 2025 17:54

sourcery-ai bot reviewed Jun 6, 2025

View reviewed changes

singer_sdk/streams/core.py Show resolved Hide resolved

singer_sdk/streams/core.py Show resolved Hide resolved

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Taps can now opt-in to emitting `ACTIVATE_VERSION` messages for streams with `FULL_TABLE` replication #2686

feat: Taps can now opt-in to emitting `ACTIVATE_VERSION` messages for streams with `FULL_TABLE` replication #2686

Uh oh!

edgarrmondragon commented Sep 24, 2024 •

edited by sourcery-ai bot

Loading

Uh oh!

codspeed-hq bot commented Sep 24, 2024 •

edited

Loading

Uh oh!

codecov bot commented Sep 24, 2024 •

edited

Loading

Uh oh!

edgarrmondragon commented Jun 3, 2025

Uh oh!

sourcery-ai bot commented Jun 3, 2025 •

edited

Loading

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

sourcery-ai bot left a comment

Uh oh!

Uh oh!

sourcery-ai bot Jun 3, 2025

Uh oh!

Uh oh!

Uh oh!

sourcery-ai bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

feat: Taps can now opt-in to emitting ACTIVATE_VERSION messages for streams with FULL_TABLE replication #2686

feat: Taps can now opt-in to emitting ACTIVATE_VERSION messages for streams with FULL_TABLE replication #2686

Uh oh!

Conversation

edgarrmondragon commented Sep 24, 2024 • edited by sourcery-ai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related

Summary by Sourcery

Summary by Sourcery

Uh oh!

codspeed-hq bot commented Sep 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #2686 will not alter performance

Summary

Uh oh!

codecov bot commented Sep 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

edgarrmondragon commented Jun 3, 2025

Uh oh!

sourcery-ai bot commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviewer's Guide

File-Level Changes

Assessment against linked issues

Possibly linked issues

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

sourcery-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sourcery-ai bot Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sourcery-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

feat: Taps can now opt-in to emitting `ACTIVATE_VERSION` messages for streams with `FULL_TABLE` replication #2686

feat: Taps can now opt-in to emitting `ACTIVATE_VERSION` messages for streams with `FULL_TABLE` replication #2686

edgarrmondragon commented Sep 24, 2024 •

edited by sourcery-ai bot

Loading

codspeed-hq bot commented Sep 24, 2024 •

edited

Loading

codecov bot commented Sep 24, 2024 •

edited

Loading

sourcery-ai bot commented Jun 3, 2025 •

edited

Loading