Versatile Data Kit 0.3

[antoniivanov]

Summary

Major features include:

Support for Kerberos Authentication provider in the Control Service

Alongside support for Oauth2, now organizations can integrate with their Kerberos infrastructure.
Users can specify Kerberos as an authentication provider for accessing VDK Control Service.

For more information on how to configure Kerberos see VDK helm documentation here

A new plugin: vdk-lineage (alpha)

VDK Lineage plugin provides lineage data (input data -> job -> output data) information for any SQL query (regardless of the database) executed using VDK and sends it to a pre-configured destination using OpenLineage standard

We also have introduced a utility command vdk marquez-server --start which starts Marquez UI locally so that lineage is visualized.

For more information check out vdk-lineage plugin documentation

Support for Kuberentes 1.23

Now VDK Control Service can work seamlessly with the newest versions of Kubernetes and make use of its features:

• VDK Control Service can now work with CronJob controller V2 (alongside V1).
• With TTL Controller, any jobs launched by VDK Control Service can be cleaned up after preconfigured time.

Users can override the VDK version of a deployed data jobs

Users can now specify the vdk version both using API or CLI when deploying a Data Job.
For example, with CLI it's as simple as vdk deploy -p <job-dir> --vdk-version old-vdk-version

This would enable canary deployments or rolling deployments of VDK.

Introducing VEP (VDK Enhancement Proposal) process and first VEP

Versatile Data Kit has a process in place for proposing and adding large changes in an efficient and consistent manner.

For more information check the process here.

We also have used the process for our first major feature change - Apache Airflow Integration

Package versions

See installation instructions here.
The versions of VDK components released under VDK 0.3 are:

Main components

vdk-control-cli
vdk-core
vdk-heartbeat
control-service

Plugins

vdk-trino
vdk-lineage
vdk-kerberos-auth
vdk-impala

What's Changed

• VEP-554: Apache Airflow Integration by @mivanov1988 in VEP-554: Apache Airflow Integration #748 and @doks5 in VEP-554: Extend Security and Permissions section #786
• airflow-provider-vdk: Initial Airflow provider structure by @gageorgiev in airflow-provider-vdk: Initial Airflow provider structure #772
• airflow-provider-vdk: Job execution status and logs method by @gageorgiev in airflow-provider-vdk: Job execution status and logs method #796
• airflow-provider-vdk: Start and cancel job execution methods by @gageorgiev in airflow-provider-vdk: Start and cancel job execution methods #778
• airflow-provider-vdk: VDKSensor initial structure by @gageorgiev in airflow-provider-vdk: VDKSensor initial structure #800
• control-service: Adopt kubernetes-client 14.0.1 by @gageorgiev in control-service: Adopt kubernetes-client 14.0.1 #761
• control-service: add kerberos auth properties to helm chart by @mrMoZ1 in [helm-chart]: add kerberos auth properties to helm chart #764
• control-service: Adopt use of the V1CronJob API by @gageorgiev in control-service: Adopt use of the V1CronJob API #767
• control-service: Bump pipelines-control-service version by @doks5 in control-service: Bump pipelines-control-service version #762
• control-service: Set TTLAfterFinished period for K8s CronJobs by @gageorgiev in control-service: Set TTLAfterFinished period for K8s CronJobs #776
• control-service: Update CHANGELOG.md by @doks5 in control-service: Update CHANGELOG.md #760
• control-service: add OAuth2 enable/disable flag by @mrMoZ1 in control-service: add OAuth2 enable/disable flag #765
• control-service: add kerberos auth provider by @mrMoZ1 in control-service: add kerberos auth provider #755
• control-service: builder job configurable security context by @mivanov1988 in control-service: builder job configurable security context #708
• control-service: configurable builder job service account by @mivanov1988 in control-service: configurable builder job service account #791
• control-service: fix builder security context by @mivanov1988 in control-service: fix builder security context #784
• control-service: fix concatAddresses NPE by @mivanov1988 in control-service: fix concatAddresses NPE #782
• control-service: fix job builder unit tests by @mivanov1988 in control-service: fix job builder unit tests #792
• control-service: fix log link to set endTime always by @tozka in control-service: fix log link to set endTime always #735
• vdk-control-cli: Adopt click version 8 by @ivakoleva in vdk-control-cli: Adopt click version 8 #770
• vdk-control-cli: set vdk version and enabled when deploying new job by @tozka in vdk-control-cli: set vdk version and enabled when deploying new job #752
• vdk-core: JobInput get_name and get_job_directory implementation by @ivakoleva in vdk-core: JobInput get_name and get_job_directory implementation #745
• vdk-core: Verify payload after pre-processing it by @YanaZhivkova in vdk-core: Verify payload after pre-processing it #777
• vdk-core: clarify run descriptions on --arguments option by @tozka in vdk-core: clarify run descriptions on --arguments option #731
• vdk-core: ensure sql args are subsituted in correct priority by @tozka in vdk-core: ensure sql args are subsituted in correct priority #749
• vdk-core: lowercase env variables are inferred as configuration by @tozka in vdk-core: lowercase env variables are inferred as configuration #751
• vdk-core: minor refactoring in managed_cursor to reduce long method by @tozka in vdk-core: minor refactoring in managed_cursor to reduce long method #803
• vdk-core: print query duration by @mrMoZ1 in vdk-core: print query duration #804
• vdk-core: refactor test to use job_path method by @tozka in vdk-core: refactor test to use job_path method #747
• vdk-core: update plugin hook diagrams by @tozka in vdk-core: update plugin hook diagrams #775
• vdk-core: Adopt click version 8.0 by @doks5 in [vdk-core][vdk-control-cli] Adopt click version 8.0 #769
• vdk-heartbeat: Fix initial job executions with specific vdk version by @YanaZhivkova in vdk-heartbeat: Fix initial job executions with specific vdk version #758
• vdk-heartbeat: Handle execution end_time not string by @doks5 in vdk-heartbeat: Handle execution end_time not string #750
• vdk-impala: unify names of templates betwen trino and impala by @tozka in vdk-impala: unify names of templates betwen trino and impala #787
• vdk-kerberos-auth: support kerberos auth for all CLI commands by @tozka in vdk-kerberos-auth: support kerberos auth for all CLI commands #774
• vdk-kerberos-auth: upgrade minikerberos and requests-kerberos to latest by @ivakoleva in vdk-kerberos-auth: upgrade minikerberos and requests-kerberos to latest #742
• vdk-lineage: introducing POC (pre-alpha) implementation by @tozka in vdk-lineage: introducing POC (pre-alpha) implementation #783
• vdk-plugins: Introduce vdk-control-api-auth plugin by @doks5 in vdk-plugins: Introduce vdk-control-api-auth plugin #801
• vdk-snowflake: Enable support for Python 3.10 by @gageorgiev in vdk-snowflake: Enable support for Python 3.10 #746
• vdk-trino: add link to template examples by @tozka in vdk-trino: add link to template examples #788
• vdk-trino: collect lineage for select/insert and rename table only by @philip-alexiev in vdk-trino: collect lineage for select/insert and rename table only #756
• vdk-trino: fix ingesting value with bool type failing by @tozka in vdk-trino: fix ingesting value with bool type failing #753
• vdk: add VDK enhancement proposal (VEP) spec template by @tozka in vdk: add VDK enhancement proposal (VEP) spec template #727
• versatile-data-kit: Update CONTRIBUTING.md with links to coding standard by @tozka in versatile-data-kit: Update CONTRIBUTING.md with links to coding standard #794

New Contributors

• @philip-alexiev made their first contribution in vdk-trino: collect lineage for select/insert and rename table only #756

Full Changelog: 0.2...0.3

---

This discussion was created from the release Versatile Data Kit 0.3.