Versatile Data Kit 1.3

[antoniivanov]

Major features include:

VDK SDK

Add vdk sql-query command (experimental)

CLI command to execute SQL query against VDK managed database.

It should replace vdk <db>-query commands.

export db_default_type=trino
vdk sql-query -q "select * from trino_table" 
id                                  memory_size_mb    num_vcpus
-------------- ----------------  -----------
50181506DB2F7               256            1
5018A2223FC32               128            1
501883404870A               256            1


vdk sql-query -o json -q "select * from trino_table" 
[
 {"id": "50181506DB2F7", "memory_size_mb": 256, "num_vcpus": 1}, 
 {"id": "5018A2223FC32", "memory_size_mb": 128, "num_vcpus": 1},
 {"id": "501883404870A", "memory_size_mb": 256, "num_vcpus": 1}
]

VDK Notebook Getting Started

Introduction to the development of VDK Jobs using Notebooks.

• Learn how to create data jobs
• Learn how to deploy data jobs

See more in https://github.com/vmware/versatile-data-kit/blob/main/projects/vdk-plugins/vdk-jupyter/getting-started.ipynb

VDK SQL Notebook Cell

VDK Errors APIs

(Relevant for plugin developers)
VDK is deprecating the user of errors.log_and_rethrow and errors.log_and_throw in favour of

errors.report(error_type, exception: BaseException)
errors.report_and_throw(exception: BaseVdkException)
errors.report_and_rethrow(error_type, exception: BaseException)

The aim is to reduce "double" logging and verbosity of logs.

Control Service

Add support for multiple jwt issuers

New properly security.oauth2.jwtIssuerUris is introduced and replaced jwtIssuerUrl

security:
   oauth2:
      
      ## [ Required if security.enabled = True ]
      ## Deprecated in favor of jwtIssuerUris.
      jwtIssuerUrl: ""
      ## [Required if security.enabled = True]
      ## Comma separated list of issuers to use.
      jwtIssuerUris: ""

Implement Webhook APIs authentication

Enable setting system or service account for Control Service webhook authentication

This introduces 2 new properties for each webhook

      authorizationServerEndpoint: ""
      authorizationRefreshToken: ""

If set they will be used when making HTTP Webhook request. If not it will fall back to the user provided authentication token

What's Changed

• control-service: add amazon rds ca certificates to data job image by @mrMoZ1 in control-service: add amazon rds ca certificates to data job image #2660
• control-service: add data job deployment dynamic property source selection mechanism by @mrMoZ1 in control-service: add data job deployment dynamic property source selection mechanism #2641
• control-service: add missing packages to secure job builder by @mivanov1988 in control-service: add missing packages to secure job builder #2662
• control-service: add support for multiple jwt issuers by @mrMoZ1 in control-service: add support for multiple jwt issuers #2628
• control-service: data jobs synchronizer initial implementation by @mivanov1988 in control-service: data jobs synchronizer initial implementation #2633
• control-service: implement Webhook APIs authentication by @mivanov1988 in control-service: implement Webhook APIs authentication #2655
• control-service: introduce data job deployment entity by @mivanov1988 in control-service: introduce data job deployment entity #2659
• control-service: webhook authentication comments by @mivanov1988 in control-service: webhook authentication comments #2663
• specs: VDK Run Logs: add detailed design for Progress tracker functionality by @antoniivanov in specs: VDK Run Logs: add detailed design for Progress tracker functionality #2647
• specs: VEP-2421 Universal Database Plugin by @Maximiliaan72 in specs: VEP-2421 Universal Database Plugin #2616
• vdk-control-api-auth: [bug-fix] authorization header base64 encoding by @doks5 in vdk-control-api-auth: [bug-fix] authorization header base64 encoding #2658
• vdk-control-cli: download keytab along with downloading job by @duyguHsnHsn in vdk-control-cli: download keytab along with downloading job  #2617
• vdk-core: add log stacktrace flag by @DeltaMichael in vdk-core: add log stacktrace flag #2648
• vdk-core: add more unit tests to for job_input_error_classfier by @DeltaMichael in vdk-core: add more unit tests to for job_input_error_classfier #2621
• vdk-core: add vdk sql-query command by @antoniivanov in vdk-core: add vdk sql-query command #2649
• vdk-core: change jobs behavior when logging unavailable by @mrMoZ1 in vdk-core: change jobs behavior when logging unavailable #2656
• vdk-core: provide more info about the machine vdk is running on by @antoniivanov in vdk-core: provide more info about the machine vdk is running on #2629
• vdk-core: remove box around exception when printed by @DeltaMichael in vdk-core: remove box around exception when printed #2700
• vdk-core: separate error logging, error reporting and error classification by @DeltaMichael in vdk-core: separate error logging, error reporting and error classification #2666
• vdk-csv: uncomment test by @duyguHsnHsn in vdk-csv: uncomment test #2704
• vdk-duckdb: update readme by @antoniivanov in vdk-duckdb: update readme #2657
• vdk-heartbeat: Add python version configuration by @doks5 in vdk-heartbeat: Add python version configuration #2675
• vdk-ipython: add support for %%vdksql magic command by @antoniivanov in vdk-ipython: add support for %%vdksql magic command #2619
• vdk-jupyter: Change root FS position for path parameter by @gageorgiev in vdk-jupyter: Change root FS position for path parameter #2550
• vdk-jupyter: Log file is not refreshed on new vdk run by @duyguHsnHsn in vdk-jupyter: Log file is not refreshed on new vdk run  #2687
• vdk-jupyter: Update ConvertJobToNotebook doc by @antoniivanov in vdk-jupyter: Update ConvertJobToNotebook doc #2702
• vdk-jupyter: VDK menu fix by @duyguHsnHsn in vdk-jupyter: VDK menu fix #2635
• vdk-jupyter: add first version of getting started page by @duyguHsnHsn in vdk-jupyter: add first version of getting started page #2516
• vdk-jupyter: add opionated jupyter extension as extra by @antoniivanov in vdk-jupyter: add opionated jupyter extension as extra #2625
• vdk-jupyter: change Create and Download default path by @duyguHsnHsn in vdk-jupyter: change Create and Download default path  #2669
• vdk-jupyter: fix arguments input size by @duyguHsnHsn in vdk-jupyter: fix arguments input size #2685
• vdk-jupyter: fix conversion by @duyguHsnHsn in vdk-jupyter: fix conversion #2699
• vdk-jupyter: improve init message by @antoniivanov in vdk-jupyter: improve init message #2688
• vdk-jupyter: pin traitlets to 5.9 by @antoniivanov in vdk-jupyter: pin traitlets to 5.9 #2674
• vdk-jupyter: rename log file by @duyguHsnHsn in vdk-jupyter: rename log file #2703
• vdk-jupyter: show status on starting new operation by @antoniivanov in vdk-jupyter: show status on starting new operation #2627
• vdk-jupyter: use npm in build.sh by @antoniivanov in vdk-jupyter: use npm in build.sh #2665
• vdk-jupyter: show status dialog on deploy and create by @antoniivanov in vdk-jupyterlab: show status dialog on deploy and create #2631
• vdk-notebook: add support for execute %%vdksql cells by @antoniivanov in vdk-notebook: add support for execute %%vdksql cells #2622
• versatile-data-kit: CONTRIBUTING.md feedback / improvements Move Developer Documentation from the Wiki to the CONTRIBUTING.md #2146 by @zverulacis in versatile-data-kit: CONTRIBUTING.md feedback / improvements #2146 #2661
• versatile-data-kit: README user testing improvements by @zverulacis in versatile-data-kit: README user testing improvements  #2654

Full Changelog: v1.2...v1.3

---

This discussion was created from the release Versatile Data Kit 1.3.