Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(tap): Utilize Joblib to run parallel streams during sync_all #2295

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

feat(tap): Utilize Joblib to run parallel streams during sync_all #2295

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
72a7ebb
added TAP_MAX_PARALLELISM_CONFIG to capabilities
BuzzCutNorman Mar 7, 2024
6720090
added TAP_MAX_PARALLELISM_CONFIG capability to Tap Class
BuzzCutNorman Mar 7, 2024
1421227
added max_parallelism property to Tap class
BuzzCutNorman Mar 7, 2024
b488a1f
added sync_one method and strated utilizing it in sync_all
BuzzCutNorman Mar 7, 2024
5101978
add option to sync_all for joblib parallel using loky to call sync_one
BuzzCutNorman Mar 7, 2024
502b36d
fixed missing console logging from stream processes
BuzzCutNorman Mar 7, 2024
73342d4
Merge branch 'main' into feat-tap-parallel-streams
edgarrmondragon May 9, 2024
3323b45
Merge branch 'main' of https://github.com/BuzzCutNorman/sdk into feat…
BuzzCutNorman Aug 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
added TAP_MAX_PARALLELISM_CONFIG capability to Tap Class
  • Loading branch information
@BuzzCutNorman
BuzzCutNorman committed Mar 7, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
commit 6720090db1c16b941bdfec5d7a4afac6247f38c3
4 changes: 4 additions & 0 deletions singer_sdk/tap_base.py
View file
Original file line number Diff line number Diff line change
@@ -24,6 +24,7 @@
from singer_sdk.helpers._util import read_json_file
from singer_sdk.helpers.capabilities import (
BATCH_CONFIG,
TAP_MAX_PARALLELISM_CONFIG,
CapabilitiesEnum,
PluginCapabilities,
TapCapabilities,
@@ -218,6 +219,9 @@ def append_builtin_config(cls: type[PluginBase], config_jsonschema: dict) -> Non
capabilities = cls.capabilities
if PluginCapabilities.BATCH in capabilities:
merge_missing_config_jsonschema(BATCH_CONFIG, config_jsonschema)
merge_missing_config_jsonschema(
TAP_MAX_PARALLELISM_CONFIG, config_jsonschema
)

# Connection and sync tests: