Airbyte v40

[thomas-gerber]

Description

Moving to Airbyte 0.40.9
Also:
Fixes #197
Fixes #288

TODO:

• Add the other sources besides the main 4
• documentation links need to be changed as the IDs are not fixed anymore
• migration strategy
• docker extension upgrade (separate PR, once the new init is released

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Migration

A simple restart will upgrade users to 0.40.9. Note that those migrated users will remain on their existing sources and Faros destination versions (until we re-enable auto-update).
This was tested the following way:

1. start an older version of CE (Airbyte v0.39.37-alpha)
2. sync Jira and GitHub data through the CLI
3. stop that older Faros, but do not delete the volumes
4. start a newer version of CE (this branch, on Airbyte v0.40.9)
5. do an incremental sync on those 2 sources

Ability to update source schema for canned sources

It will work if the starting version of CE is after this merge. Older users of Faros still have to recreate the sources and connections.
This was tested the following way:

1. start a newer version of CE
2. configure the GitHub source and sync it
3. in the connection replication settings, refresh source schema
4. one sees all the streams we haven't configured as new, and all existing streams as having been modified
5. enable pull_request_stats and save
6. sync again, and that new stream catches up

A note on sync mode

We either use Incremental with append_dedup if incremental is available source-side, full_refresh with overwrite otherwise. Note that we don't fully abide by the spec, as we do NOT delete the data for that stream in the overwrite mode. Unfortunately, there is no sync mode that describes that behavior ATM.

Note that all existing streams are marked as modified only because we didn't pass along the json schema during canned setup, which is now captured for the first time

Checklist

• Have you checked to there aren't other open Pull Requests for the same update/change?
• Have you lint your code locally before submission?
• Have you added an explanation of what your changes do and why you'd like us to include them?
• Have you successfully run tests with your changes locally?

[ogusak]

If we plan on upgrading k8s deploy - this needs to be changed as well

[thomas-gerber]

If we plan on upgrading k8s deploy - this needs to be changed as well

Fixed