Skip to content

pw.io.airbyte.read: allow users to override connector dependency versions #201

@zxqfd555

Description

@zxqfd555

Is your feature request related to a problem? Please describe.
Airbyte connectors available as PyPI packages often declare their dependencies without version upper bounds or exact pins. This means that installing a connector at any point in time may pull in a different (and potentially incompatible) version of a transitive dependency compared to what was installed previously.

The correct long-term fix is upstream: connector maintainers should pin their dependencies properly. However, that fix may be slow to arrive — or may never come for some connectors. In the meantime, users of pw.io.airbyte.read have no way to stabilize their environment without forking or monkey-patching the connector package itself.

A concrete recent example: airbyte-cdk removed MessageRepresentationAirbyteTracedErrors in a non-backward-compatible release, which broke airbyte-source-github and several other connectors. Because no version pin was in place, this cascaded immediately to the users and integration tests.

Describe the solution you'd like
Add an optional dependency_overrides parameter to pw.io.airbyte.read that allows users to specify additional pip constraints to apply when installing the connector. For example:

pw.io.airbyte.read(
    ...,
    dependency_overrides=["airbyte-cdk==X.Y.Z"],
)

This would allow users to:

  • Pin a known-good version of a broken transitive dependency
  • Unblock a pipeline immediately, without waiting for an upstream fix
  • Reproduce a stable environment on repeated runs

Describe alternatives you've considered

  • Waiting for upstream connector fixes — not always timely, and sometimes the fix never comes.
  • Vendoring / forking the connector — disproportionate effort for what should be a simple version pin.
  • Full environment pinning at the project level — works in some setups, but conflicts with how pw.io.airbyte.read dynamically installs connectors.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions