-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Description
Is your feature request related to a problem? Please describe.
Airbyte connectors available as PyPI packages often declare their dependencies without version upper bounds or exact pins. This means that installing a connector at any point in time may pull in a different (and potentially incompatible) version of a transitive dependency compared to what was installed previously.
The correct long-term fix is upstream: connector maintainers should pin their dependencies properly. However, that fix may be slow to arrive — or may never come for some connectors. In the meantime, users of pw.io.airbyte.read have no way to stabilize their environment without forking or monkey-patching the connector package itself.
A concrete recent example: airbyte-cdk removed MessageRepresentationAirbyteTracedErrors in a non-backward-compatible release, which broke airbyte-source-github and several other connectors. Because no version pin was in place, this cascaded immediately to the users and integration tests.
Describe the solution you'd like
Add an optional dependency_overrides parameter to pw.io.airbyte.read that allows users to specify additional pip constraints to apply when installing the connector. For example:
pw.io.airbyte.read(
...,
dependency_overrides=["airbyte-cdk==X.Y.Z"],
)This would allow users to:
- Pin a known-good version of a broken transitive dependency
- Unblock a pipeline immediately, without waiting for an upstream fix
- Reproduce a stable environment on repeated runs
Describe alternatives you've considered
- Waiting for upstream connector fixes — not always timely, and sometimes the fix never comes.
- Vendoring / forking the connector — disproportionate effort for what should be a simple version pin.
- Full environment pinning at the project level — works in some setups, but conflicts with how
pw.io.airbyte.readdynamically installs connectors.