v0.43.1

v0.43.1 (2024-12-10)

🐛 Fixes

• #2807 Allow developers to set RESTStream.http_method

v0.43.0

v0.43.0 (2024-12-10)

Highlights

This release introduces a few improvements and changes to the SDK that should make it easier to work with SQL taps, REST streams, and targets.

• A new SQL tap class attribute exclude_streams allows developers to exclude certain database schemas by default from discovery. This can speed up discovery and avoid issues when mapping types from database-internal schemas.

• Discovery for SQL taps should be about 35% faster now, thanks to using SQLAlchemy 2.0+ reflection features.

• A new target built-in setting process_activate_version_messages lets users disable processing of ACTIVATE_VERSION messages.

• Support for other content-types in REST streams with the new RESTStream.payload_as_json attribute. This should make it easier to work with APIs that expect non-JSON payloads, such as XML.

• The RESTStream.rest_method attribute is deprecated in favor of RESTStream.http_method.

Important

This release drops support for Python 3.8. Please make sure to upgrade your tap or target to support Python 3.9+.

Note

SQL taps now require SQLAlchemy 2.0+.

✨ New

• #2482 Allow SQL tap developers to auto-skip certain schemas from discovery
• #2784 Added a new built-in setting activate_version for targets to optionally disable processing of ACTIVATE_VERSION messages
• #2780 Numeric values are now parsed as decimal.Decimal in REST and GraphQL stream responses
• #2775 Log a stream's bookmark (if it's avaiable) when its sync starts
• #2703 Targets now emit record count from the built-in batch file processor
• #2774 Accept a maxLength limit for VARCHARs
• #2769 Add versioning-strategy to dependabot config of Cookiecutter templates
• #2765 The last received Singer message is now logged when the target fails
• #2762 Support other content-types in REST streams

🐛 Fixes

• #2790 Ensure the required global folder tap settings are merged into the concrete implementation settings
• #2785 Use FS-specific listdir in folder tap
• #2778 The path of the offending field is now printed for config validation errors
• #2770 Respect standard Singer stream metadata table-key-properties, replication-key and forced-replication-method
• #2755 Safely compare UUID replication keys with state bookmarks -- Thanks @nikzavada!

⚙️ Under the Hood

• #2805 Rename setting activate_version to process_activate_version_messages
• #2788 Fail early if input files to --catalog or --state do not exist
• #2781 Added a class method to instantiate SQLToJSONSchema from the tap configuration
• #2566 Standardize on JSON Schema Draft 2020-12 to validate stream schemas
• #2751 Dropped support for Python 3.8

⚡ Performance Improvements

• #2793 Improved discovery performance for SQL taps

📚 Documentation Improvements

• #2796 Document how to configure nested stream maps values with environment variables in Meltano

📦 Packaging changes

• #2797 SQL taps now require SQLAlchemy 2.0+

v0.42.1

v0.42.1 (2024-11-11)

🐛 Fixes

• #2756 Safely compare UUID replication keys with state bookmarks -- Thanks @nikzavada!

Full Changelog: v0.42.0...v0.42.1

v0.42.0

v0.42.0 (2024-11-11)

Highlights

• For SQL target developers, there is now a public and official API for overriding how JSON schema types are mapped into SQL types. Let us know if there's any improvements we can make to it to better suit your use case!
• The name of the stream is now available in stream maps as __stream_name__, including in the context of __alias__ expressions.
• This is last minor version to support Python 3.8, which reached its EOL on 2024-10-07, the next release will only support Python 3.9+.

✨ New

• #2742 Update dependencies in templates
• #2732 SQL target developers can now more easily override the mapping from JSON schema to SQL column type
• #2730 Added SQLConnector.prepare_primary_key for target to implement for custom table primary key adaptation
• #2488 Nested schema properties can now be defined as nullable
• #2518 Python 3.13 is officially supported
• #2637 Environment variables are now parsed for boolean, integer, array and object setting values
• #2699 Stream name can now be accessed in stream maps -- Thanks @holly-evans!
• #2712 JSON schema title is now supported in configuration and stream properties
• #2707 Bumped simpleeval to 1.0
• #2701 Stream name can now be accessed in __alias__ context of stream maps -- Thanks @holly-evans!

🐛 Fixes

• #2741 datetime.datetime instances in stream maps are now correctly mapped to date-time JSON schema strings
• #2727 Object and array JSON types are now handled before primitive types when converting them to SQL types
• #2723 JSON schema union types are no longer conformed into boolean values

⚙️ Under the Hood

• #2743 Deprecate passing file paths to plugin and stream initialization

📚 Documentation Improvements

• #2745 Document the current release process
• #2717 Update Meltano commands in examples

📦 Packaging changes

• #2736 Skip simpleeval 1.0.1
• #2716 Stopped testing with SQLAlchemy 1.4
• #2714 Remove constraint on urllib3

v0.41.0

v0.41.0 (2024-10-02)

Highlights

• It's easier now for SQL tap developers to customize the mapping from SQL column types to JSON schema. See the guide for details.

✨ New

• #2667 Support stream aliasing of BATCH messages via stream maps -- Thanks @ReubenFrankel!
• #2651 SQL taps now emit schemas with maxLength when applicable
• #2618 Developers can now more easily override the mapping from SQL column type to JSON schema

🐛 Fixes

• #2697 All HTTP timeout exceptions are now retried in REST and GraphQL streams
• #2683 A clear error message is now emitted when flattening is enabled but flattening_max_depth is not set
• #2665 Mapped datetime values are now typed as date-time strings in the schema message -- Thanks @gregkoutsimp!
• #2663 Properties dropped using None or __NULL__ in stream maps are now also removed from the schema required array

⚙️ Under the Hood

• #2696 Use tox without installing Poetry explicitly in workflows
• #2654 Added a generic FileStream (still in active development!)
• #2695 Update dependencies in templates
• #2661 Drop support for Python 3.8 in templates
• #2670 Deprecated Faker class in stream maps
• #2666 Remove non-functional record-flattening capability -- Thanks @ReubenFrankel!
• #2652 Renamed SQLConnector.type_mapping to SQLConnector.sql_to_jsonschema
• #2647 Use future warnings.deprecated decorator

📚 Documentation Improvements

• #2658 Added more versions when stuff changed or was added

📦 Packaging changes

• #2694 Removed unused backport importlib_resources dependency in tap template
• #2664 Added a constraint on setuptools <= 70.3.0 to fix incompatibility with some dependencies

v0.40.0

v0.40.0 (2024-09-02)

Highlights

• Targets now emit basic metrics. There's nothing to do on your end, they're enabled by default!
• The Faker class is available in stream maps context. Note that like the faker instance added in v0.35.0, this requires installing the faker library in the tap/target environment. For example, with tap-example[faker].

✨ New

• #2486 Emit target metrics
• #2567 A new schema_is_valid built-in tap test validates stream schemas against the JSON Schema specification
• #2598 Stream map expressions now have access to the Faker class, rather than just a faker instance
• #2549 Added a default user agent for REST and GraphQL taps

🐛 Fixes

• #2613 Mismatch between timezone-aware and naive datetimes in start date and bookmarks is now correctly handled

⚙️ Under the Hood

• #2628 Use context manager to read gzip batch files
• #2619 Default to UTC when parsing dates without a known timezone
• #2603 Backwards-compatible identifier quoting in fully qualified names
• #2601 Improved SQL identifier (de)normalization
• #2599 Remove pytest-durations dependency from testing and use native pytest option --durations
• #2597 Mark pagination classes with @override decorator
• #2596 Made auth_headers and auth_params of APIAuthenticatorBase simple instance attributes instead of decorated properties

📚 Documentation Improvements

• #2639 Documented versions where fake and Faker objects were added to the stream maps context
• #2629 Reference get_starting_timestamp in incremental replication guide
• #2604 Update project sample links
• #2595 Documented examples of stream glob expressions and property aliasing

📦 Packaging changes

• #2640 Remove upper constraint on faker extra

v0.39.1

v0.39.1 (2024-08-07)

• PyPI 🐍

🐛 Fixes

• #2589 Make sink assertion compatible with stream maps -- Thanks @JCotton1123!
• #2592 Fixed typos in --about plain text output
• #2580 Date fields are now properly serialized as ISO dates, i.e. "YYYY-MM-DD"
• #2583 Quote add-column-ddl with column starting with _ (underscore) based on engine -- Thanks @haleemur!
• #2582 DDL for adding a column now uses a SQLAlchemy-compiled clause to apply proper quoting -- Thanks @haleemur!
• #2418 Check replication method instead of key to determine if a SQL stream is sorted

⚙️ Under the Hood

• #2520 Remove unused dependencies pendulum and python-dateutil

📚 Documentation Improvements

• #2579 Documented support for packaging semantic type in contributing guide

v0.39.0

v0.39.0 (2024-07-27)

• PyPI 🐍

Highlights

• The json module can now be used within stream maps

  stream_maps:
    animals:
      id: id
      data: "json.dumps({'description': description, 'verified': verified, 'views': views, 'created_at': created_at})"
      __else__: null

• The crypto dependencies PyJWT and cryptography are now optional and can be installed by requesting the jwt extra, i.e. singer-sdk[jwt]. These are only currently required for JWT authentication, but if your package depends on any of these, make sure the dependency is explicit.

• The pendulum and python-dateutil dependencies are no longer used by this library and will be removed in a future release. If your package relies on any of them, make sure the dependency is explicit.

• We're working on supporting msgspec for more performant IO, so keep an eye out for that 😉.

✨ New

• #2432 Developers can now customize the default logging configuration for their taps/targets by adding default_logging.yml to their package
• #2531 The json module is now avaiable to stream maps -- Thanks @grigi!
• #2529 Stream sync context is now available to all instances methods as a Stream.context attribute

🐛 Fixes

• #2554 Use mapped stream aliases when handling ACTIVATE_VERSION messages in the base target class
• #2526 Moved up the supported Python versions in the Markdown output of --about

⚙️ Under the Hood

• #2564 Make SQLSink a generic with a SQLConnector type parameter
• #2540 Implement abstract serialize_message for Singer writers
• #2259 Centralize JSON SerDe into helper functions -- Thanks @BuzzCutNorman!
• #2525 Make PyJWT and cryptography dependencies optional
• #2528 Moved class-level attributes to the top in REST tap template
• #2132 Limit internal usage of pendulum

📚 Documentation Improvements

• #2557 Document that get_starting_timestamp requires setting a non-null replication_key
• #2556 Reference state partitioning in stream partitioning page
• #2536 Prepare for RTD addons migration
• #2535 Added more intersphinx links to Python and Faker docs
• #2530 Explained how the request URL is generated from url_base, path and the sync context
• #2527 Updated the footer
• #2506 Fixed a typo in the stream maps docs

v0.38.0

v0.38.0 (2024-06-17)

✨ New

• #2433 Tap developers can now disable HTTP redirects
• #2426 Added an optional GitHub workflow to publish to PyPI with trusted publishers

🐛 Fixes

• #2438 Null replication values are now handled when incrementing bookmarks
• #2431 Updated cookiecutter VSCode launch.json to use debugpy
• #2421 An error message is now logged every time schema validation fails for any record

⚙️ Under the Hood

• #2455 Use parent datetime.datetime class in type conforming checks
• #2453 Change to return type of utc_now from pendulum.DateTime to datetime.datetime

📚 Documentation Improvements

• #2449 Add a short guide on defining a configuration schema
• #2436 Documented how context fields are passed to a child stream
• #2435 Using an empty list for __key_properties__ to disable a stream primary keys is now recommended as an alternative to null

v0.37.0

v0.37.0 (2024-04-29)

Highlights

• You can reference your streams with glob patterns in stream map configuration:

  stream_maps:
    # This will affect all streams with names starting with 'my_prefix_'
    "my_prefix_*":
      id: id
      first_name: first_name
      email:     # drop the PII field from RECORD and SCHEMA messages
      email_domain: email.split('@')[-1]
      email_hash: md5(config['hash_seed'] + email)
      __else__: null

✨ New

• #2389 JSON schema keyword allOf is now supported
• #1888 Added support for glob patterns in source stream names -- Thanks @DouweM!
• #2345 PropertiesList can now behave as an iterable -- Thanks @ReubenFrankel!

🐛 Fixes

• #2352 Removed unnecessary and problematic column caching -- Thanks @raulbonet!
• #2375 Added sensitive: true to password settings in templates
• #2301 Unmapped sub-fields in object-type fields are now no longer dropped when the field declares additionalProperties
• #2348 Added a condition to the No schema for record field warning -- Thanks @tobiascadee!
• #2342 Avoid failing if VSCode IDE config is not requested for target and mapper cookiecutter templates -- Thanks @ReubenFrankel!
• #2331 Allow importlib-resources >=6.3.2

⚙️ Under the Hood

• #2205 Added a jwt package extra, but the cryptography and jwt dependencies are still installed by default for now

📚 Documentation Improvements

• #2326 Documented BATCH as a default plugin capability -- Thanks @ReubenFrankel!