Skip to content

Clean up for workspace scan tables #2788

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 72 commits into from
Oct 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
69ddb8f
Add history install
JCZuurmond Sep 25, 2024
02d8a30
Add history install to `WorkspaceContext`
JCZuurmond Sep 25, 2024
26314ea
Run history installation in create UCX catalog
JCZuurmond Sep 25, 2024
786f097
Fix test
JCZuurmond Sep 25, 2024
0180114
Add unit test for creating UCX history schema and table
JCZuurmond Sep 25, 2024
1e7cdac
Test history installation to create history schema
JCZuurmond Sep 25, 2024
50fb6e5
Format
JCZuurmond Sep 25, 2024
a3cd1c5
Create UCX catalog on `MockWorkspaceContext`
JCZuurmond Sep 25, 2024
60389b5
Remove `Record` data class and table creation
JCZuurmond Sep 26, 2024
8aeba40
Make test more robust
JCZuurmond Sep 26, 2024
5161074
Use `SchemaDeployer` to deploy schema
JCZuurmond Sep 26, 2024
4fe1de6
Add `WorkflowRun` dataclass
JCZuurmond Sep 26, 2024
11c3cb9
Make test more robust
JCZuurmond Sep 26, 2024
1ad8bb4
Create `workflow_runs` table
JCZuurmond Sep 26, 2024
68145f9
Rename `HistoryInstallation` to `ProgressTrackingInstaller`
JCZuurmond Sep 26, 2024
9c08d39
Rename progress tracking schem to 'multiworkspace'
JCZuurmond Sep 26, 2024
2f0e019
Move WorkflowRun to separate module
JCZuurmond Sep 26, 2024
302fc1d
Add workflow run recorder
JCZuurmond Sep 26, 2024
8054949
Add start time to workflow
JCZuurmond Sep 26, 2024
c58d9ed
Expect start time on record
JCZuurmond Sep 26, 2024
af119bb
Add attempt to workflow run
JCZuurmond Sep 26, 2024
01b438f
Add job name and start time to task parameters
JCZuurmond Sep 26, 2024
1c9b31d
Add `WorkflowRunRecorder` to `RuntimeContext`
JCZuurmond Sep 26, 2024
48a1a86
Expect workflow start time on initialisation
JCZuurmond Sep 26, 2024
c7d8dff
Record workflow run
JCZuurmond Sep 26, 2024
f967194
Add record workflow run task to migration progress workflow
JCZuurmond Sep 26, 2024
b84e60a
Format
JCZuurmond Sep 26, 2024
9c25c06
Fix reference to catalog
JCZuurmond Sep 26, 2024
fdb643d
Fix TODO
JCZuurmond Sep 26, 2024
34d44e5
Remove start time on Workflow
JCZuurmond Sep 27, 2024
6cf4c97
Record UCX internal workflow name
JCZuurmond Sep 27, 2024
097727c
Remove job name from job parameters
JCZuurmond Sep 27, 2024
cfb44ad
Remove WorkflowRun.status
JCZuurmond Sep 27, 2024
4c8242c
Add missing UCX catalog parameter
JCZuurmond Sep 27, 2024
4721364
Test progress tracking installer creates workflow runs table
JCZuurmond Sep 27, 2024
e249dae
Log when workflow run table is not found
JCZuurmond Sep 27, 2024
01f207a
Test workflow run recorder records a record
JCZuurmond Sep 27, 2024
ea672e1
Make finished at timestamp more precise
JCZuurmond Sep 27, 2024
ee194ea
Fix UNKNOWN typo
JCZuurmond Sep 27, 2024
c60cc31
Test workflow run values
JCZuurmond Sep 27, 2024
34975bf
Use non-escaping characters in UCX test catalog
JCZuurmond Sep 27, 2024
2d83096
Test workflow runs is populated in the during real migration progress
JCZuurmond Sep 27, 2024
da602c9
Use workflow from named parameters at init
JCZuurmond Sep 27, 2024
950d2f0
Add workspace id as task parameter
JCZuurmond Sep 27, 2024
4be9454
Pass new parameters to main
JCZuurmond Sep 27, 2024
08aef94
Remove WorkflowRuns.run_as
JCZuurmond Sep 27, 2024
d33986c
Get workspace id from job parameters
JCZuurmond Sep 27, 2024
44f5f66
Set test local ucx catalog on installation context config
JCZuurmond Sep 27, 2024
965a8fc
Rename progress tracking installer to installation
JCZuurmond Sep 27, 2024
e4e84fc
Add ProgressTrackingInstallation to MockInstallationContext
JCZuurmond Sep 27, 2024
7070bba
Format
JCZuurmond Sep 27, 2024
fe943a0
Add unit test for workflow runs
JCZuurmond Sep 27, 2024
64d8e43
Be saver with finished at test
JCZuurmond Sep 27, 2024
ec2aada
Format README
JCZuurmond Sep 27, 2024
3280e85
Explain workflow runs in the README
JCZuurmond Sep 27, 2024
b3146b1
Fix `workflow_start_time` should be `start_time`
JCZuurmond Sep 27, 2024
9502991
Format
JCZuurmond Sep 27, 2024
57b7d8c
Get workspace id from workspace client
JCZuurmond Sep 27, 2024
c8de2c9
Fix typo's
JCZuurmond Oct 1, 2024
32c1d5e
Add type hint for `WorkflowRunRecorder`
JCZuurmond Oct 2, 2024
931dfeb
Improve comment
JCZuurmond Oct 2, 2024
92fb2fb
Improve docstring
JCZuurmond Oct 2, 2024
5adfc6a
Remove redundant argument
JCZuurmond Oct 2, 2024
b3f9be2
Add assert message
JCZuurmond Oct 2, 2024
d127cec
Get test workspace id from workspace client
JCZuurmond Oct 2, 2024
d9a793e
Add dummy test to increase coverage
JCZuurmond Oct 2, 2024
b03e269
Add assert message for failing workflow
JCZuurmond Oct 2, 2024
19757da
Format
JCZuurmond Oct 2, 2024
ec2ff9c
Fix types
JCZuurmond Oct 2, 2024
9337be2
Set max wait on run assessment workflow to 25 minutes
JCZuurmond Oct 2, 2024
22ad1b8
Merge branch 'main' into feat/add-workspace-scans-table
JCZuurmond Oct 2, 2024
ed84fee
Merge branch 'main' into feat/add-workspace-scans-table
JCZuurmond Oct 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions tests/integration/progress/test_workflows.py
Original file line number Diff line number Diff line change
@@ -1,27 +1,29 @@
from datetime import timedelta
import datetime as dt

from databricks.sdk.errors import NotFound, InvalidParameterValue
from databricks.sdk.retries import retried

from ..conftest import MockInstallationContext


@retried(on=[NotFound, InvalidParameterValue], timeout=timedelta(minutes=12))
@retried(on=[NotFound, InvalidParameterValue], timeout=dt.timedelta(minutes=12))
def test_running_real_migration_progress_job(installation_ctx: MockInstallationContext) -> None:
"""Ensure that the migration-progress workflow can complete successfully."""
installation_ctx.workspace_installation.run()

# The assessment workflow is a prerequisite for migration-progress: it needs to successfully complete before we can
# test the migration-progress workflow.
installation_ctx.deployed_workflows.run_workflow("assessment")
assert installation_ctx.deployed_workflows.validate_step("assessment")
workflow = "assessment"
installation_ctx.deployed_workflows.run_workflow(workflow, max_wait=dt.timedelta(minutes=25))
assert installation_ctx.deployed_workflows.validate_step(workflow), f"Workflow failed: {workflow}"

# After the assessment, a user (maybe) installs the progress tracking
installation_ctx.progress_tracking_installation.run()

# Run the migration-progress workflow until completion.
installation_ctx.deployed_workflows.run_workflow("migration-progress-experimental")
assert installation_ctx.deployed_workflows.validate_step("migration-progress-experimental")
workflow = "migration-progress-experimental"
installation_ctx.deployed_workflows.run_workflow(workflow)
assert installation_ctx.deployed_workflows.validate_step(workflow), f"Workflow failed: {workflow}"

# Ensure that the migration-progress workflow populated the `workflow_runs` table.
query = f"SELECT 1 FROM {installation_ctx.ucx_catalog}.multiworkspace.workflow_runs"
Expand Down
37 changes: 37 additions & 0 deletions tests/unit/contexts/test_workflow_task.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import datetime as dt
import pytest

from databricks.labs.ucx.config import WorkspaceConfig
from databricks.labs.ucx.contexts.workflow_task import RuntimeContext
from databricks.labs.ucx.progress.workflow_runs import WorkflowRunRecorder


class MockWorkspaceClient:
"""Mock workspace client"""

def get_workspace_id(self) -> int:
return 123456789


@pytest.mark.parametrize(
"attribute, klass",
[
("workflow_run_recorder", WorkflowRunRecorder),
],
)
def test_context_attribute_class(tmp_path, attribute, klass) -> None:
"""Increase coverage with these tests"""
config = WorkspaceConfig("str")
named_parameters = {
"workflow": "test",
"job_id": "123",
"parent_run_id": "456",
"start_time": dt.datetime.now(tz=dt.timezone.utc).replace(microsecond=0).isoformat(),
}

ctx = RuntimeContext(named_parameters=named_parameters).replace(
config=config, sql_backend=None, workspace_client=MockWorkspaceClient()
)

assert hasattr(ctx, attribute)
assert isinstance(getattr(ctx, attribute), klass)
Loading