Release v0.3.1

What's Changed

• 🐛 fix sometimes wrong type check when writing df by @danielgafni in #55

Full Changelog: v0.3.0...v0.3.1

Release v0.3.1a0

Full Changelog: v0.3.0...v0.3.1a0

Release v0.3.0

✨ What's Changed

• feat: add lazyframe support for parquet and delta by @ion-elgreco in #52. sink_df_to_path method can be used to implement LazyFrame sinking
• cloud_storage_options config field added. Supports passing values with dagster.EnvVar, for example:

PolarsDeltaIOManager(cloud_storage_options={"aws_secret_access_key": EnvVar("AWS_SECRET_ACCESS_KEY")})

• removed legacy pyarrow parquet reader

💥 Breaking Changes

• minimal compatible polars version has been increased to 0.20.0
• BasePolarsUPathIOManager - renamed dump_df_to_path to write_df_to_path

🤝 New Contributors

• @ion-elgreco made their first contribution in #52. Thank you for your work!

Full Changelog: v0.2.2...v0.3.0

Release v0.3.0a0

Testing release for LazyFrame sink support

Full Changelog: v0.2.2...v0.3.0a0

Release v0.2.2

🐛 Bug Fixes

• Due to an issue with UPathIOManager, PolarsDeltaIOManager did not apply correct partition filtering when loading a single mapped partition of an asset with native DeltaLake partitioning.
  A workaround for this issue has been included.
• Also, it's now possible to load all asset partitions at once with native DeltaLake partitioning, by specifying a type annotation like pl.DataFrame or pl.LazyFrame. No partitions filtering will be done in this case.

What's Changed

• 🐛 DeltaLake - fix loading single partitions, implement loading all partitions at once by @danielgafni in #51

Full Changelog: v0.2.1...v0.2.2

Release v0.2.1

What's Changed

• 🐛 properly overwrite DeltaLake partitions by @danielgafni in #50

Full Changelog: v0.2.0...v0.2.1

Release v0.2.0

💥 Breaking Changes

• Rename BigQueryIOManager -> PolarsBigqueryIOManager

What's Changed

• ⭐ skip BigQueryIOManager upload if output is None or empty by @danielgafni in #46
• 📝 💥 prepare for merge into Dagster project by @danielgafni in #48
• ✨ automatically filter partitions in the IOManagers by @danielgafni in #49

Full Changelog: v0.1.5...v0.2.0

Release v0.1.5

What's Changed

• 👷 add required qa_success CI step by @danielgafni in #42
• ✨ support storing arbitrary metadata in PolarsDeltaIOManager by @danielgafni in #44
• 👷 fix matrix packages installation in CI by @danielgafni in #45

Full Changelog: v0.1.4...v0.1.5

Release v0.1.4

What's Changed

The new native polars Parquet reader is going to be used with polars >= 0.19.4.
The old, pyarrow reader can be enabled in PolarsParquetIOManager config.

• 👷 add manual CI trigger by @danielgafni in #39
• 🐛 fix missing pyarrow.dataset import for pyarrow 13.0.0 by @seandavi in #38
• ⚡ use native parquet reader by @danielgafni in #41

New Contributors

• @seandavi made their first contribution in #38

Full Changelog: v0.1.3...v0.1.4

Release v0.1.3

What's Changed

• 👷 remove MacOS tests in CI/CD by @danielgafni in #34
• 🔊 log Delta Lake table version when loading it & make sure time-traveling works by @danielgafni in #36

Full Changelog: v0.1.2...v0.1.3