Clean up for workspace scan tables #2788

JCZuurmond · 2024-10-02T15:45:40Z

Had some dangling commits for #2754

github-actions · 2024-10-02T16:10:26Z

❌ 4/5 passed, 1 failed, 8m21s total

❌ test_running_real_migration_progress_job: AssertionError: Workflow failed: assessment (5m57.324s)

AssertionError: Workflow failed: assessment
assert False
 +  where False = validate_step('assessment')
 +    where validate_step = <databricks.labs.ucx.installer.workflows.DeployedWorkflows object at 0x7fbf74167430>.validate_step
 +      where <databricks.labs.ucx.installer.workflows.DeployedWorkflows object at 0x7fbf74167430> = <tests.integration.conftest.MockInstallationContext object at 0x7fbf74164910>.deployed_workflows
[gw2] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python
10:01 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.wmG7/config.yml) doesn't exist.
10:01 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
10:01 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
10:01 INFO [databricks.labs.ucx.install] Fetching installations...
10:01 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
10:01 DEBUG [tests.integration.conftest] Waiting for clusters to start...
10:03 DEBUG [tests.integration.conftest] Waiting for clusters to start...
10:03 INFO [databricks.labs.ucx.install] Installing UCX v0.38.1+9120241003100323
10:03 INFO [databricks.labs.ucx.install] Creating ucx schemas...
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migration-progress-experimental
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
10:03 INFO [databricks.labs.ucx.install] Creating dashboards...
10:03 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
10:03 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
10:03 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
10:03 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
10:03 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
10:03 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
10:03 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
10:03 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
10:03 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
10:03 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
10:03 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
10:03 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
10:03 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
10:03 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
10:03 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
10:03 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.wmG7/README for the next steps.
10:03 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/338545089711221
10:03 INFO [databricks.labs.ucx.installer.workflows] Started assessment job: https://DATABRICKS_HOST#job/338545089711221/runs/346021885675194
10:03 DEBUG [databricks.labs.ucx.installer.workflows] Waiting for completion of assessment job: https://DATABRICKS_HOST#job/338545089711221/runs/346021885675194
10:07 INFO [databricks.labs.ucx.installer.workflows] Completed assessment job run 346021885675194 with state: RunResultState.CANCELED (Run cancelled by user)
10:07 INFO [databricks.labs.ucx.installer.workflows] Completed assessment job run 346021885675194 duration: 0:03:15.370000 (2024-10-03 10:03:55.112000+00:00 thru 2024-10-03 10:07:10.482000+00:00)
10:07 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/338545089711221
10:01 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.wmG7/config.yml) doesn't exist.
10:01 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
10:01 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
10:01 INFO [databricks.labs.ucx.install] Fetching installations...
10:01 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
10:01 DEBUG [tests.integration.conftest] Waiting for clusters to start...
10:03 DEBUG [tests.integration.conftest] Waiting for clusters to start...
10:03 INFO [databricks.labs.ucx.install] Installing UCX v0.38.1+9120241003100323
10:03 INFO [databricks.labs.ucx.install] Creating ucx schemas...
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migration-progress-experimental
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
10:03 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
10:03 INFO [databricks.labs.ucx.install] Creating dashboards...
10:03 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
10:03 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
10:03 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
10:03 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
10:03 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
10:03 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
10:03 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
10:03 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
10:03 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
10:03 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
10:03 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
10:03 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
10:03 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
10:03 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
10:03 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
10:03 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.wmG7/README for the next steps.
10:03 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/338545089711221
10:03 INFO [databricks.labs.ucx.installer.workflows] Started assessment job: https://DATABRICKS_HOST#job/338545089711221/runs/346021885675194
10:03 DEBUG [databricks.labs.ucx.installer.workflows] Waiting for completion of assessment job: https://DATABRICKS_HOST#job/338545089711221/runs/346021885675194
10:07 INFO [databricks.labs.ucx.installer.workflows] Completed assessment job run 346021885675194 with state: RunResultState.CANCELED (Run cancelled by user)
10:07 INFO [databricks.labs.ucx.installer.workflows] Completed assessment job run 346021885675194 duration: 0:03:15.370000 (2024-10-03 10:03:55.112000+00:00 thru 2024-10-03 10:07:10.482000+00:00)
10:07 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/338545089711221
10:07 INFO [databricks.labs.ucx.install] Deleting UCX v0.38.1+9120241003100323 from https://DATABRICKS_HOST
10:07 ERROR [databricks.labs.ucx.install] Check if /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.wmG7 is present
[gw2] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

_{Running from acceptance #6312}

nfx

lgtm

JCZuurmond · 2024-10-03T10:55:25Z

Integration test failure is unrelated to changes in the PR

JCZuurmond added 30 commits September 27, 2024 12:18

Add history install

69ddb8f

Add history install to WorkspaceContext

02d8a30

Run history installation in create UCX catalog

26314ea

Fix test

786f097

Add unit test for creating UCX history schema and table

0180114

Test history installation to create history schema

1e7cdac

Format

50fb6e5

Create UCX catalog on MockWorkspaceContext

a3cd1c5

Remove Record data class and table creation

60389b5

Make test more robust

8aeba40

Use SchemaDeployer to deploy schema

5161074

Add WorkflowRun dataclass

4fe1de6

Make test more robust

11c3cb9

Create workflow_runs table

1ad8bb4

Rename HistoryInstallation to ProgressTrackingInstaller

68145f9

Rename progress tracking schem to 'multiworkspace'

9c08d39

Move WorkflowRun to separate module

2f0e019

Add workflow run recorder

302fc1d

Add start time to workflow

8054949

Expect start time on record

c58d9ed

Add attempt to workflow run

af119bb

Add job name and start time to task parameters

01b438f

Add WorkflowRunRecorder to RuntimeContext

1c9b31d

Expect workflow start time on initialisation

48a1a86

Record workflow run

c7d8dff

Add record workflow run task to migration progress workflow

f967194

Format

b84e60a

Fix reference to catalog

9c25c06

Fix TODO

fdb643d

Remove start time on Workflow

34d44e5

JCZuurmond added 15 commits September 27, 2024 12:18

Format

9502991

Get workspace id from workspace client

57b7d8c

Fix typo's

c8de2c9

Add type hint for WorkflowRunRecorder

32c1d5e

Improve comment

931dfeb

Improve docstring

92fb2fb

Remove redundant argument

5adfc6a

Add assert message

b3f9be2

Get test workspace id from workspace client

d127cec

Add dummy test to increase coverage

d9a793e

Add assert message for failing workflow

b03e269

Format

19757da

Fix types

ec2ff9c

Set max wait on run assessment workflow to 25 minutes

9337be2

Merge branch 'main' into feat/add-workspace-scans-table

22ad1b8

JCZuurmond added tech debt chores and design flaws internal this pull request won't appear in release notes labels Oct 2, 2024

JCZuurmond self-assigned this Oct 2, 2024

JCZuurmond requested a review from a team as a code owner October 2, 2024 15:45

JCZuurmond had a problem deploying to account-admin October 2, 2024 15:45 — with GitHub Actions Failure

nfx approved these changes Oct 2, 2024

View reviewed changes

Merge branch 'main' into feat/add-workspace-scans-table

ed84fee

JCZuurmond had a problem deploying to account-admin October 3, 2024 10:00 — with GitHub Actions Failure

JCZuurmond mentioned this pull request Oct 3, 2024

Fix missing arguments to WorkflowLinter #2809

Merged

JCZuurmond enabled auto-merge October 3, 2024 10:56

nfx disabled auto-merge October 3, 2024 11:22

nfx merged commit 6cc819a into main Oct 3, 2024
5 of 6 checks passed

nfx deleted the feat/add-workspace-scans-table branch October 3, 2024 11:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Clean up for workspace scan tables #2788

Clean up for workspace scan tables #2788

Uh oh!

JCZuurmond commented Oct 2, 2024

Uh oh!

github-actions bot commented Oct 2, 2024 •

edited

Loading

Uh oh!

nfx left a comment

Uh oh!

JCZuurmond commented Oct 3, 2024

Uh oh!

Uh oh!

Uh oh!

Clean up for workspace scan tables #2788

Clean up for workspace scan tables #2788

Uh oh!

Conversation

JCZuurmond commented Oct 2, 2024

Uh oh!

github-actions bot commented Oct 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nfx left a comment

Choose a reason for hiding this comment

Uh oh!

JCZuurmond commented Oct 3, 2024

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 2, 2024 •

edited

Loading