Versatile Data Kit 0.11

[antoniivanov]

Major features include:

Introduce data quality checks POC (for scd1 template)

Allow quality checks to be made before the data is inserted into the target table.
Currently, the checks done on the processing step are not covering if the semantics of the data is correct. Therefore, bad data could went into the target table which could be unwanted behavior.

Example:

    def sample_check_true(tmp_table_name):
        return False if "bad" in tmp_table_name  elseTrue 

    template_args["check"] = sample_check 
    job_input.execute_template(
        template_name="load/dimension/scd1",
        template_args=template_args,
    )

Jobs Query API (GraphQL) wildcard matching filter for team and job names

When querying information about jobs now users of the Jobs QUery API can use wildcard matches :
wildcard matching for example *search* in graphQl filters for job name and team name as well as before exact matching of search strings

Provide User Agent when using VDK CLI

Users are looking to be able to determine where requests originated from when analyzing and browsing the telemetry data about VDK Control Service usage.

export VDK_CONTROL_SERVICE_USER_AGENT = foo 

or in config.ini

[vdk]
vdk_control_service_user_agent=foo

If not set it would default to "vdk-control-cli/{version} ({os.name}; {sys.platform})" + {python version}

New plugin: vdk-notebook

A new VDK plugin that supports running data jobs which consists of .ipynb files. You can see VDK Notebook plugin page for more information.

vdk-ipython

This extension introduces a magic command for Jupyter. The command enables the user to load job_input for his current data job and use it freely while working with Jupyter.
You can see VDK ipython plugin page for more information.

What's Changed

• control service: remove deprecated dependency by @murphp15 in Control service: remove deprecated dependency  #1589
• control-service: Remove dependency on old docker image which is not needed by @murphp15 in Control-service: Remove dependency on old docker image which is not needed #1548
• control-service: Fixed data job status in case of OOM by @mivanov1988 in Fixed data job status in case of OOM #1586
• control-service: base-job-image: automatic image cleanup by @mivanov1988 in base-job-image: automatic image cleanup #1636
• control-service: Cronjob API backwards compatibility by @doks5 in control-service: Cronjob API backwards compatibility #1580
• control-service: Fix release step in pipeline by @murphp15 in control-service: Fix release step in pipeline #1550
• control-service: Remove duplicated CICD job runs by @mivanov1988 in control-service: Remove duplicated CICD job runs #1596
• control-service: cleanup tests to ease testing on control service (v2) by @murphp15 in control-service: cleanup tests to ease testing on control service (v2) #1607
• control-service: cleanup tests to ease testing on control service by @murphp15 in control-service: cleanup tests to ease testing on control service #1604
• control-service: configurable job initContainer resources by @mivanov1988 in control-service: configurable job initContainer resources #1599
• control-service: graphql revert part of the wildcard filter matching by @mrMoZ1 in control-service: graphql revert part of the wildcard filter matching #1615
• control-service: handle init container OOM by @mivanov1988 in control-service: handle init container OOM #1658
• control-service: integration tests refactoring by @mivanov1988 in control-service: integration tests refactoring #1562
• control-service: java 17 by @murphp15 in control-service: java 17 #1439
• control-service: only delete file in test path by @murphp15 in control-service: only delete file in test path #1545
• control-service: remove deprecated classes in codebase. by @murphp15 in control-service: remove deprecated classes in codebase. #1611
• control-service: remove old kerberous test dependency by @murphp15 in control-service: remove old kerberous test dependency #1539
• control-service: upgrade gradle. by @murphp15 in control-service: upgrade gradle. #1543
• control-service: use latest docker image by @murphp15 in control-service: use latest docker image #1538
• frontend: data-pipelines (create initial repository documentation #1) bundle root by @ivakoleva in frontend: data-pipelines (#1) bundle root #1626
• frontend: data-pipelines (remove duplicate code-of-conduct.md #2) lib bundle root by @ivakoleva in frontend: data-pipelines (#2) lib bundle root #1629
• frontend: data-pipelines (vdk/cicd: initial integration with Gitlab CI #3) lib bundle sources by @ivakoleva in frontend: data-pipelines (#3) lib bundle sources #1630
• frontend: data-pipelines (vdk/cicd: initial integration with Gitlab CI #4) ui bundle root by @ivakoleva in frontend: data-pipelines (#4) ui bundle root #1631
• frontend: data-pipelines (vdk: update contributing and readme #5) ui bundle sources by @ivakoleva in frontend: data-pipelines (#5) ui bundle sources #1632
• frontend: open source shared components package by @DeltaMichael in frontend: open source shared components package #1618
• job-base-image-secure: remove unused parameter from publication script by @mivanov1988 in job-base-image-secure: remove unused parameter from publication script #1650
• job-builder: address docker image vulnerabilities by @mivanov1988 in job-builder: address docker image vulnerabilities #1523
• job-builder: fix ci/cd steps by @mivanov1988 in job-builder: fix ci/cd steps #1555
• job-builder: introduced secure base-job-image by @mivanov1988 in job-builder: introduced secure base-job-image #1546
• job-builder: remove toybox from the base job image by @mivanov1988 in job-builder: remove toybox from the base job image #1552
• vdk-cicd: cleanup cicd rules by @murphp15 in vdk-cicd: cleanup cicd rules #1554
• vdk-cicd: during a scheduled run publish_artifacts and release shouldn't run by @murphp15 in vdk-cicd: during a scheduled run publish_artifacts and release shouldn't run #1551
• vdk-control-service: fix null dereferences by @dakodakov in vdk-control-service: fix null dereferences #1512
• vdk-control-service: fix possible NPE by @dakodakov in vdk-control-service: fix possible NPE #1522
• vdk-control-service: potential resource leak fixes by @dakodakov in vdk-control-service: potential resource leak fixes #1513
• vdk-core: track configuration sensitivity by @DeltaMichael in vdk-core: track configuration sensitivity #1579
• vdk-frontend: docker image for running end to end tests in gitlab by @murphp15 in vdk-frontend: docker image for running end to end tests in gitlab #1563
• vdk-frontend: include readmes for the data-pipelines folders by @murphp15 in vdk-frontend: include readmes for the data-pipelines folders #1598
• vdk-frontend: open source readmes by @murphp15 in vdk-frontend: open source readmes #1537
• vdk-impala: Introduce checks for scd1 template by @sbuldeev in vdk-impala: Introduce checks for scd1 template #1472
• vdk-jobs-troubleshooting: Run troubleshooting server as deamon thread by @dakodakov in vdk-jobs-troubleshooting: Run troubleshooting server as deamon thread #1499
• vdk-jupyter: add create job command to jupyter by @duyguHsnHsn in vdk-jupyter: add create job command to jupyter #1581
• vdk-jupyter: add download job command to jupyter by @duyguHsnHsn in vdk-jupyter: add download job command to jupyter #1492
• vdk-jupyter: create iPython extension by @duyguHsnHsn in vdk-jupyter: create iPython extension #1483
• vdk-jupyter: fixes on tsconfig and bad file naming by @duyguHsnHsn in vdk-jupyter: fixes on tsconfig and bad file naming #1594
• vdk-jupyter: improve error handling on the UI by @duyguHsnHsn in vdk-jupyter: improve error handling on the UI #1528
• vdk-jupyter: make VEP more accessible and informative by @duyguHsnHsn in vdk-jupyter: make VEP more accessible and informative  #1635
• vdk-jupyter: modify the way we read notebooks in notebook plugin by @duyguHsnHsn in vdk-jupyter: modify the way we read notebooks in notebook plugin #1520
• vdk-jupyter: modify the way we work with notebooks in notebook plugin by @duyguHsnHsn in vdk-jupyter: modify the way we work with notebooks in notebook plugin  #1564
• vdk-jupyter: ui end-to-end testing by @duyguHsnHsn in vdk-jupyter: ui end-to-end testing #1617
• vdk-jupyter: vdk-notebook README improvements by @duyguHsnHsn in vdk-jupyter: vdk-notebook README improvements #1642
• vdk-meta-jobs: Better error message for misspelled job name by @gageorgiev in vdk-meta-jobs: Better error message for misspelled job name  #1592
• vdk-snowflake: upgrade to Python 3.11 by @tozka in vdk-snowflake: upgrade to Python 3.11 #1609
• vdk-spec: cleanup template by @murphp15 in vdk-spec: cleanup template #1518
• vdk-spec: describe package publishing by @murphp15 in vdk-spec: describe package publishing #1536
• vdk-spec: folder structure by @murphp15 in vdk-spec: folder structure #1525
• vdk-spec: remove api section because the frontend will have no impact on the api by @murphp15 in vdk-spec: remove api section because the frontend will have no impact on the api #1524
• vdk-spec: summary, glossary, motivation by @murphp15 in vdk-spec: summary, glossary, motivation #1521
• vdk-test-utils: add cli_assert_output_contains by @tozka in vdk-test-utils: add cli_assert_output_contains #1540
• versatile-data-kit: update changelog instructions by @tozka in vdk: update changelog instructions #1541
• versatile-data-kit: Meta Job example by @gageorgiev in versatile-data-kit: Meta Job example #1640
• versatile-data-kit: copyright notice year update by @ivakoleva in versatile-data-kit: copyright notice year update #1634
• versatile-data-kit: git pre-commit hooks config by @ivakoleva in versatile-data-kit: git pre-commit hooks config #1625
• versatile-data-kit: remove unnecessary changelog files by @tozka in versatile-data-kit: remove unnecessary changelog files #1549
• versatile-data-kit: update readme with mail list by @tozka in versatile-data-kit: update readme with mail list #1561

New Contributors

• @sbuldeev made their first contribution in vdk-impala: Introduce checks for scd1 template #1472
• @gary-tai made their first contribution in Update README #1535
• @DeltaMichael made their first contribution in vdk-core: track configuration sensitivity #1579

Full Changelog: v0.10...v0.11

---

This discussion was created from the release Versatile Data Kit 0.11.