v0.36.1

v0.36.1 (2024-03-13)

🐛 Fixes

• #2310 Limited importlib-resources to < 6.2 due to breaking changes
• #2288 Bumped min joblib dependency to >=1.3.0 and replaced deprecated parallel_backend with parallel_config -- Thanks @BuzzCutNorman!
• #2281 The state_partition_context dictionary is now correctly interpolated in the error message when duplicate partitions/contexts are detected in the input state
• #2274 Test workflow now fails for any Python matrix job failure in cookiecutter template -- Thanks @ReubenFrankel!

v0.36.0

v0.36.0 (2024-02-26)

✨ New

• #2241 JSON schema keywords such as maxLength are now supported in StringType, IntegerType and NumberType JSON schema helpers
• #2263 Nested settings are now documented in the output of --about --format=markdown
• #2248 Targets now accept a batch_size_rows setting to configure how many rows are loaded in each record batch -- Thanks @BuzzCutNorman!

🐛 Fixes

• #2258 Database disconnects are now handled via SQLAlchemy pool_pre_ping parameter

⚙️ Under the Hood

• #2220 Deprecated singer_sdk.authenticators.BasicAuthenticator in favor of requests.auth.HTTPBasicAuth

v0.35.2

v0.35.2 (2024-02-19)

🐛 Fixes

• #2252 Null type is now correctly appended as {"type": "null"} to oneOf types

v0.35.1

v0.35.1 (2024-02-15)

🐛 Fixes

• #2243 Flattening level of record is now forced to match the provided flattened schema -- Thanks @joaopamaral!
• #2245 Instances of oneOf are now handled by null-appending logic
• #2242 Hard and soft deletes for handling ACTIVATE_VERSION messages now use the same WHERE clause -- Thanks @vitoravancini!
• #2232 Test workflow job now fails for unsupported Python versions in cookiecutter templates -- Thanks @ReubenFrankel!
• #2225 SQL columns that are non-nullable but not required (i.e. not part of a primary key) are now not included in the "required": [...] array of the discovered JSON schema

📚 Documentation Improvements

• #2239 Linked reference docs to source code
• #2231 Added an example implemetation of JSON schema validation that uses fastjsonschema
• #2219 Added reference docs for tap & target testing helpers

v0.35.0

v0.35.0 (2024-02-02)

Highlights

• The package now acts a proper pytest plugin, so you may need to remove pytest_plugins = ("singer_sdk.testing.pytest_plugin",) from your tests/conftest.py file for test fixtures to be loaded correctly.
• Dropped support for Python 3.7 (EOL 2023-06-27).
• Dropped direct dependency on pytz. If your package uses pytz in any capacity, you may need to add it as an explicit dependency to your project.
• Stream maps can now generate fake data! See the docs for more details.

✨ New

• #2208 Allow users to disable schema validation in targets
• #2170 Generate fake data with stream maps -- Thanks @ReubenFrankel!
• #937 Support validating configuration for any tap with a dynamic catalog
• #2144 Support fanning out parent record into multiple child contexts/syncs
• #1918 End RESTStream pagination if an empty page is received

🐛 Fixes

• #2203 Fix serialization of arbitrary objects (e.g. ObjectId from mongoDB) during flattening -- Thanks @dgawlowsky!
• #2200 Quote column names in INSERT statement
• #2195 Include empty schemas directory in REST tap cookiecutter
• #2187 Replace use of deprecated jsonschema._RefResolver with recommended referencing library
• #2184 Reduce amount of unnecessary whitespace in Singer output
• #2183 Ensure .vscode directory is included when requested in cookiecutters and avoid failing if it does not exist
• #2180 Limit supported Python versions in --about output to existing ones
• #2108 Log sink name when an unhandled error occurs during setup
• #2158 Fix pytest plugin declaration so it can be used without requiring defining pytest_plugins in conftest.py
• #2105 Default handling of ACTIVATE_VERSION messages to soft deletes and add new SQLConnector.delete_old_versions method

⚙️ Under the Hood

• #2189 Use functools.lru_cache instead of the stale memoization library (#1981)
• #2188 Remove unused logger parameter from private catalog helper functions
• #2143 Drop support for Python 3.7
• #2157 Remove pytz dependency and use datetime.timezone.utc instead of pytz.UTC where possible
• #2136 Create interface for schema validation in sinks, and implement it for python-jsonschema -- Thanks @BuzzCutNorman!
• #2130 Allow loading stream schemas from importlib.resources.abc.Traversable types

📚 Documentation Improvements

• #2204 Document supported package extras
• #2186 Call out minimum recommended cookiecutter version
• #2168 Explain Progress is not resumable if interrupted in docs FAQ
• #2140 Update auth caching example to use functools.cached_property

v0.34.1

v0.34.1 (2023-12-19)

🐛 Fixes

• #2118 Output JSONPath expression with match count message -- Thanks @mjsqu!
• #2107 Respect forced replication method when retrieving state
• #2094 Use nulls_first when available to order NULL results in incremental SQL streams

⚙️ Under the Hood

• #1733 Test with Python 3.12 🐍
• #2095 Use CursorResult.mappings() in SQL streams
• #2092 Use datetime.fromisoformat in other places
• #2090 Explicitly use T iso date separator

📚 Documentation Improvements

• #2111 Fix broken requests documentation links -- Thanks @mjsqu!

v0.34.0

v0.34.0 (2023-12-05)

Hightlights

• Date-time parsing in target sinks is now significantly faster, thanks to @BuzzCutNorman!
• Experimental support for Parquet encoding in BATCH messages, thanks to @jamielxcarter!

✨ New

• #2044 Add Parquet as a batch encoding option -- Thanks @jamielxcarter!
• #768 Better error messages when config validation fails
• #1854 Make stream logger a child of the tap logger

🐛 Fixes

• #2076 Make a explicit dependency on python-dateutil

⚙️ Under the Hood

• #2070 Parse dates with datetime.fromisoformat/backports.datetime_fromisoformat -- Thanks @BuzzCutNorman!
• #2056 Break up TapTestClassFactory._annotate_test_class into simpler methods
• #2058 Implement a SingerWriter class in singer_sdk.io_base and use it to emit Singer messages -- Thanks @BuzzCutNorman!
• #2061 Simplify target template file names with post_gen_project.py hook -- Thanks @vicmattos!
• #2060 Simplify tap template file names with post_gen_project.py hook -- Thanks @vicmattos!

📚 Documentation Improvements

• #2039 Add 404 page with sphinx-notfound-page
• #2037 Add flattening configuration examples

Full Changelog: v0.33.1...v0.34.0

v0.33.1

v0.33.1 (2023-11-08)

🐛 Fixes

• #2035 Retry all 5xx status codes -- Thanks @asamasoma!

v0.33.0

v0.33.0 (2023-10-12)

Hightlights

• Support for simple comprehension expression in stream maps: https://sdk.meltano.com/en/v0.33.0/stream_maps.html#compound-expressions
• Performance improvements for stream maps: expressions are now parsed once and evaluated multiple times.

---

✨ New

• #1999 Log JSONPath match count at the INFO level
• #1779 Cache SQL columns and schemas
• #2003 Add ability to do list comprehensions in stream map expressions -- Thanks @haleemur!
• #2018 Drop Python 3.7 support in cookiecutter templates -- Thanks @visch!

🐛 Fixes

• #2006 Parse record time_extracted into datetime.datetime instance
• #1996 Respect nullability of leaf properties when flattening schema
• #1844 Safely skip parsing record field as date-time if it is missing in schema
• #1885 Map record field to a JSON object type
• #2015 Ensure default property is passed to SCHEMA messages -- Thanks @prakharcode!

📚 Documentation Improvements

• #2017 Document support for comprehensions in stream maps

v0.32.0

v0.32.0 (2023-09-22)

Highlights

• Initial experimental support for different loading methods in targets. Currently, the overwrite method works by dropping and recreating tables in SQL targets. Methods upsert and append are still expected to be implemented by the target.

✨ New

• #1893 Standard configurable load methods
• #1861 SQLTap connector instance shared with streams -- Thanks @BuzzCutNorman!

🐛 Fixes

• #1954 Missing begin()s related to SQLAlchemy 2.0 -- Thanks @andyoneal!
• #1951 Ensure SQL streams are sorted when a replication key is set
• #1949 Retry SQLAlchemy engine creation for adapters without JSON SerDe support
• #1939 Handle decimal.Decimal instances in flattening
• #1927 Handle replication key not found in stream schema -- Thanks @mjsqu!
• #1977 Fix hanging downstream tests in tap-postgres
• #1970 Warn instead of crashing when schema helpers cannot append null to types

⚡ Performance Improvements

• #1962 Ensure raw_schema in stream mapper is immutable

📚 Documentation Improvements

• #1925 Add viztracer command for testing targets -- Thanks @mjsqu!