v1.6

[antoniivanov]

Major features include:

vdk-oracle database plugin

A new oracle plugin can be used to execute queries against Oracle DB in both thick and thin mode.
Ingesting data is now supported including with automatic shema inference.

To see more information check the vdk-oracle plugin documentation

vdk-structlog

Various enhancements in VDK-Structlog, including syslog handler support, log level parsing, and configuration updates

Check out more about vdk-structlog in its documentation

VDK Ingestion into Vector Database for RAG initiative started

With the rise in popularity of LLMs and RAG we see VDK as a core component to getting the data where we need it to be. VDK's strengths are ETL tasks. We see that its very well suited to populating the databases needed for RAG.

For more information check out the VEP

What's Changed

• control-service: Add support for vdk version by @doks5 in control-service: Add support for vdk version #2943
• control-service: base job image - update README.md by @mivanov1988 in control-service: base job image - update README.md #2971
• control-service: fix job builder secure by @mivanov1988 in control-service: fix job builder secure #2976
• control-service: getting all jobs should now be consistent by @mrMoZ1 in control-service: getting all jobs should now be consistent #2978
• control-service: oracle support for vdk-oracle plugin by @mivanov1988 in control-service: oracle support for vdk-oracle plugin #3042
• control-service: write resources to db by @mrMoZ1 in control-service: write resources to db #2897
• documentation: Update to READMEs: Add a monthly download badge by @chrfoyer in documentation: Update to READMEs: Add a monthly download badge #2983
• examples: add Confluence data retrieval example Data Job by @yonitoo in examples: add Confluence data retrieval example Data Job #3040
• frontend: [SC-2632] Save Executions grid state by @hzhristova in frontend: [SC-2632] Save Executions grid state #2991
• specs: update Vector DB Ingestion VEP by @antoniivanov in specs: update Vector DB Ingestion VEP #3033
• vdk-control-cli: Modify how data is written to job config.ini by @doks5 in vdk-control-cli: Modify how data is written to job config.ini #2982
• vdk-core: Short job summary by @gageorgiev in vdk-core: Short job summary #3038
• vdk-core: replace deprecated imp module with importlib by @yonitoo in vdk-core: replace deprecated imp module with importlib #2981
• vdk-core: replace report_and_rethrow by @DeltaMichael in vdk-core: replace report_and_rethrow #3056
• vdk-impala: handle decorate operation errors by @DeltaMichael in vdk-impala: handle decorate operation errors #2975
• vdk-jupyter: e2e tests fix by @duyguHsnHsn in vdk-jupyter: e2e tests fix #2966
• vdk-jupyter: update npm packages by @antoniivanov in vdk-jupyter: update npm packages #3043
• vdk-kerberos-auth: fix dependency issue by @murphp15 in vdk-kerberos-auth: fix dependency issue #2969
• vdk-license: change year by @duyguHsnHsn in vdk-license: change year  #2987
• vdk-lineage: pin sqllineage version by @duyguHsnHsn in vdk-lineage: pin sqllineage version #3005
• vdk-oracle: add new config options (host,port,sid,thick_mode_lib_dir) by @antoniivanov in vdk-oracle: add new config options (host,port,sid,thick_mode_lib_dir) #3032
• vdk-oracle: escape special chars in column names by @DeltaMichael in vdk-oracle: escape special chars in column names #3045
• vdk-oracle: fixes in tests and passing secrets by @antoniivanov in vdk-oracle: fixes in tests and passing secrets  #3031
• vdk-oracle: set thik mode default to true by @antoniivanov in vdk-oracle: set thik mode default to true #3037
• vdk-oracle: support secrets to connect to database by @DeltaMichael in vdk-oracle: support secrets to connect to database #2961
• vdk-oracle: support thick mode by default by @DeltaMichael in vdk-oracle: support thick mode by default #2970
• vdk-plugins: replace report_and_rethrow by @DeltaMichael in vdk-plugins: replace report_and_rethrow #3057
• vdk-specs: Ingestion into vector db for RAG use by @murphp15 in vdk-specs: Ingestion into vector db for RAG use #3004
• vdk-structlog: Document log level module config var by @gageorgiev in vdk-structlog: Document log level module config var #2984
• vdk-structlog: add syslog handler by @duyguHsnHsn in vdk-structlog: add syslog handler #2985
• vdk-structlog: filter vdk_step_name and vdk_step_type correctly by @DeltaMichael in vdk-structlog: filter vdk_step_name and vdk_step_type correctly #2968
• vdk-structlog: rename structlog configs by @DeltaMichael in vdk-structlog: rename structlog configs #3002
• vdk-structlog: support parse log level module and logger config granularity by @yonitoo in vdk-structlog: support parse log level module and logger config granularity #2980
• vdk-structlog: update README.md by @DeltaMichael in vdk-structlog: update README.md #3047
• vdk-structlog: update VEP-2448 to reflect current state of development by @DeltaMichael in vdk-structlog: update VEP-2448 to reflect current state of development #3035

New Contributors

• @chrfoyer made their first contribution in documentation: Update to READMEs: Add a monthly download badge #2983

Full Changelog: v1.5...v1.6

---

This discussion was created from the release v1.6.