Skip to content

Fix config-remote-sync writes through scalar YAML parents - #7323

Draft
ilyakuz-db wants to merge 4 commits into
mainfrom
ilyakuz-db/fix-config-sync-scalar-parents
Draft

Fix config-remote-sync writes through scalar YAML parents#7323
ilyakuz-db wants to merge 4 commits into
mainfrom
ilyakuz-db/fix-config-sync-scalar-parents

Conversation

@ilyakuz-db

@ilyakuz-db ilyakuz-db commented Nov 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Changes

Handle null YAML ancestors when bundle config-remote-sync --save adds nested fields, and skip changes beneath variable references at any ancestor depth. Preserve unrelated edits, skip reporting, and comments on implicit nulls, explicit null, and ~.

Variable-reference decisions stay in resolve. Null materialization uses yaml.Node in patch, where each edit sees earlier edits to the file. The retry converts only actual nulls, refuses anchors and aliases, and rejects sequence creation that would require inventing preceding elements. No new dyn dependency is introduced.

Why

Sync could fail with scalar can not be a container when an addition traversed a null YAML node or a variable reference above the immediate parent. Applying multiple additions beneath the same null must preserve all of them.

Tests

  • All 93 cases across 47 config-remote-sync acceptance directories pass, including both DMS variants. Coverage includes null ancestors, target overrides, numeric map keys, retained comments and inherited values, repeat sync, and deep variable-backed additions, replacements, and removals.
  • The original variable-reference fixture and golden output pass unchanged. The extended scenarios fail on the original production code with the scalar-container error; the strengthened null-comment fixture also fails against the previous PR revision because it drops the comments.
  • Configsync package tests, golangci-lint, formatting, whitespace checks, and git diff --check pass. Focused unit coverage checks scalar/anchor/sequence guards and root/sequence-item comments.
  • Validation used a temporary local YAML v3.0.5 override; go.mod and go.sum are unchanged. The exact v3.0.6 pin, full CLI suite, and real-workspace execution remain untested.

This PR was written by Codex.

@github-actions github-actions Bot added the DABs DABs related issues label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Commit: 4820d19c2 | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

4 tests: regression=4 unreleased=0 coverage=0 failing=0
test branch main (44f2eda) latest (v1.23.0)
TestAccept/bundle/config-remote-sync/null_parent/DMS= ✅ ❌ ❌
TestAccept/bundle/config-remote-sync/null_parent/DMS=true ✅ ❌ ❌
TestAccept/bundle/config-remote-sync/variable_reference_parent/DMS= ✅ ❌ ❌
TestAccept/bundle/config-remote-sync/variable_reference_parent/DMS=true ✅ ❌ ❌
TestAccept/bundle/config-remote-sync/null_parent/DMS=: branch ✅ | main (44f2eda) ❌ | latest (v1.23.0) ❌

main (44f2eda):

acceptance_test.go:1342: Diff:
        --- bundle/config-remote-sync/null_parent/output.txt
        +++ /tmp/TestAcceptbundleconfig-remote-syncnull_parentDMS=3134522284/002/output.txt
        @@ -6,50 +6,8 @@
         Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
         
         >>> [CLI] bundle config-remote-sync --save
        -Detected changes in 1 resource(s):
        +Error: failed to generate YAML files: failed to apply change to file [TEST_TMP_DIR]/databricks.yml for a field resources.jobs.my_job.job_clusters[1].new_cluster.custom_tags.123: failed to apply change: op add /resources/jobs/my_job/job_clusters/1/new_cluster/custom_tags/123: unexpected yaml node: scalar can not be a container
         
        -Resource: resources.jobs.my_job
        -  job_clusters[job_cluster_key='nested'].new_cluster.custom_tags['123']: add
        -  job_clusters[job_cluster_key='nested'].new_cluster.custom_tags['team']: add
        -  job_clusters[job_cluster_key='shared'].new_cluster.autoscale.max_workers: add
        -  job_clusters[job_cluster_key='shared'].new_cluster.custom_tags['123']: add
        -  job_clusters[job_cluster_key='shared'].new_cluster.custom_tags['team']: add
        -  job_clusters[job_cluster_key='shared'].new_cluster.init_scripts: replace
        -  job_clusters[job_cluster_key='shared'].new_cluster.spark_env_vars['ENV']: add
        -
        -
        -
        ->>> diff.py databricks.yml.backup databricks.yml
        ---- databricks.yml.backup
        -+++ databricks.yml
        -@@ -12,9 +12,18 @@
        -             node_type_id: [NODE_TYPE_ID]
        -             custom_tags: # Keep this comment when adding tags.
        --            spark_env_vars: null # Keep the environment comment.
        --            init_scripts: ~
        --            autoscale: ~ # Keep the autoscale comment.
        -+              "123": numeric key
        -+              team: analytics
        -+            spark_env_vars: # Keep the environment comment.
        -+              ENV: dev
        -+            init_scripts:
        -+              - workspace:
        -+                  destination: /Workspace/init.sh
        -+            autoscale: # Keep the autoscale comment.
        -+              max_workers: 2
        -         - job_cluster_key: nested
        --          new_cluster: null # Keep the cluster comment through both tag additions.
        -+          new_cluster: # Keep the cluster comment through both tag additions.
        -+            custom_tags:
        -+              "123": numeric key
        -+              team: analytics
        -       tasks:
        -         - task_key: main
        -
        ->>> [CLI] bundle config-remote-sync --save
        -No changes detected.
        -
        -
         >>> [CLI] bundle destroy --auto-approve
         The following resources will be deleted:
           delete resources.jobs.my_job
        @@ -57,3 +14,5 @@
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default
         
         Destroy: 1 deleted
        +
        +Exit code: 1

latest (v1.23.0): same output as main (44f2eda)

TestAccept/bundle/config-remote-sync/null_parent/DMS=true: branch ✅ | main (44f2eda) ❌ | latest (v1.23.0) ❌

main (44f2eda):

acceptance_test.go:1342: Diff:
        --- bundle/config-remote-sync/null_parent/output.txt
        +++ /tmp/TestAcceptbundleconfig-remote-syncnull_parentDMS=true816465558/002/output.txt
        @@ -6,50 +6,8 @@
         Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
         
         >>> [CLI] bundle config-remote-sync --save
        -Detected changes in 1 resource(s):
        +Error: failed to generate YAML files: failed to apply change to file [TEST_TMP_DIR]/databricks.yml for a field resources.jobs.my_job.job_clusters[1].new_cluster.custom_tags.123: failed to apply change: op add /resources/jobs/my_job/job_clusters/1/new_cluster/custom_tags/123: unexpected yaml node: scalar can not be a container
         
        -Resource: resources.jobs.my_job
        -  job_clusters[job_cluster_key='nested'].new_cluster.custom_tags['123']: add
        -  job_clusters[job_cluster_key='nested'].new_cluster.custom_tags['team']: add
        -  job_clusters[job_cluster_key='shared'].new_cluster.autoscale.max_workers: add
        -  job_clusters[job_cluster_key='shared'].new_cluster.custom_tags['123']: add
        -  job_clusters[job_cluster_key='shared'].new_cluster.custom_tags['team']: add
        -  job_clusters[job_cluster_key='shared'].new_cluster.init_scripts: replace
        -  job_clusters[job_cluster_key='shared'].new_cluster.spark_env_vars['ENV']: add
        -
        -
        -
        ->>> diff.py databricks.yml.backup databricks.yml
        ---- databricks.yml.backup
        -+++ databricks.yml
        -@@ -12,9 +12,18 @@
        -             node_type_id: [NODE_TYPE_ID]
        -             custom_tags: # Keep this comment when adding tags.
        --            spark_env_vars: null # Keep the environment comment.
        --            init_scripts: ~
        --            autoscale: ~ # Keep the autoscale comment.
        -+              "123": numeric key
        -+              team: analytics
        -+            spark_env_vars: # Keep the environment comment.
        -+              ENV: dev
        -+            init_scripts:
        -+              - workspace:
        -+                  destination: /Workspace/init.sh
        -+            autoscale: # Keep the autoscale comment.
        -+              max_workers: 2
        -         - job_cluster_key: nested
        --          new_cluster: null # Keep the cluster comment through both tag additions.
        -+          new_cluster: # Keep the cluster comment through both tag additions.
        -+            custom_tags:
        -+              "123": numeric key
        -+              team: analytics
        -       tasks:
        -         - task_key: main
        -
        ->>> [CLI] bundle config-remote-sync --save
        -No changes detected.
        -
        -
         >>> [CLI] bundle destroy --auto-approve
         The following resources will be deleted:
           delete resources.jobs.my_job
        @@ -57,3 +14,5 @@
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default
         
         Destroy: 1 deleted
        +
        +Exit code: 1

latest (v1.23.0):

acceptance_test.go:1342: Diff:
        --- bundle/config-remote-sync/null_parent/output.txt
        +++ /tmp/TestAcceptbundleconfig-remote-syncnull_parentDMS=true1153206960/002/output.txt
        @@ -6,50 +6,8 @@
         Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
         
         >>> [CLI] bundle config-remote-sync --save
        -Detected changes in 1 resource(s):
        +Error: failed to generate YAML files: failed to apply change to file [TEST_TMP_DIR]/databricks.yml for a field resources.jobs.my_job.job_clusters[1].new_cluster.custom_tags.123: failed to apply change: op add /resources/jobs/my_job/job_clusters/1/new_cluster/custom_tags/123: unexpected yaml node: scalar can not be a container
         
        -Resource: resources.jobs.my_job
        -  job_clusters[job_cluster_key='nested'].new_cluster.custom_tags['123']: add
        -  job_clusters[job_cluster_key='nested'].new_cluster.custom_tags['team']: add
        -  job_clusters[job_cluster_key='shared'].new_cluster.autoscale.max_workers: add
        -  job_clusters[job_cluster_key='shared'].new_cluster.custom_tags['123']: add
        -  job_clusters[job_cluster_key='shared'].new_cluster.custom_tags['team']: add
        -  job_clusters[job_cluster_key='shared'].new_cluster.init_scripts: replace
        -  job_clusters[job_cluster_key='shared'].new_cluster.spark_env_vars['ENV']: add
        -
        -
        -
        ->>> diff.py databricks.yml.backup databricks.yml
        ---- databricks.yml.backup
        -+++ databricks.yml
        -@@ -12,9 +12,18 @@
        -             node_type_id: [NODE_TYPE_ID]
        -             custom_tags: # Keep this comment when adding tags.
        --            spark_env_vars: null # Keep the environment comment.
        --            init_scripts: ~
        --            autoscale: ~ # Keep the autoscale comment.
        -+              "123": numeric key
        -+              team: analytics
        -+            spark_env_vars: # Keep the environment comment.
        -+              ENV: dev
        -+            init_scripts:
        -+              - workspace:
        -+                  destination: /Workspace/init.sh
        -+            autoscale: # Keep the autoscale comment.
        -+              max_workers: 2
        -         - job_cluster_key: nested
        --          new_cluster: null # Keep the cluster comment through both tag additions.
        -+          new_cluster: # Keep the cluster comment through both tag additions.
        -+            custom_tags:
        -+              "123": numeric key
        -+              team: analytics
        -       tasks:
        -         - task_key: main
        -
        ->>> [CLI] bundle config-remote-sync --save
        -No changes detected.
        -
        -
         >>> [CLI] bundle destroy --auto-approve
         The following resources will be deleted:
           delete resources.jobs.my_job
        @@ -57,3 +14,5 @@
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default
         
         Destroy: 1 deleted
        +
        +Exit code: 1
        
FAIL
TestAccept/bundle/config-remote-sync/variable_reference_parent/DMS=: branch ✅ | main (44f2eda) ❌ | latest (v1.23.0) ❌

main (44f2eda):

acceptance_test.go:1342: Diff:
        --- bundle/config-remote-sync/variable_reference_parent/output.txt
        +++ /tmp/TestAcceptbundleconfig-remote-syncvariable_reference_parentD2522664960/002/output.txt
        @@ -6,64 +6,8 @@
         === Edit fields under complex variables
         
         === Detect and save all changes
        -Detected changes in 1 resource(s):
        +Error: failed to generate YAML files: failed to apply change to file [TEST_TMP_DIR]/databricks.yml for a field resources.jobs.my_job.job_clusters[1].new_cluster.custom_tags.added: failed to apply change: op add /resources/jobs/my_job/job_clusters/1/new_cluster/custom_tags/added: unexpected yaml node: scalar can not be a container
         
        -Resource: resources.jobs.my_job
        -  job_clusters[job_cluster_key='shared'].new_cluster.init_scripts[0].workspace.destination: skip
        -  job_clusters[job_cluster_key='shared'].new_cluster.spark_conf['spark.executor.memory']: skip
        -  job_clusters[job_cluster_key='variable_cluster'].new_cluster.custom_tags['added']: skip
        -  job_clusters[job_cluster_key='variable_cluster'].new_cluster.custom_tags['changed']: skip
        -  job_clusters[job_cluster_key='variable_cluster'].new_cluster.custom_tags['removed']: skip
        -  job_clusters[job_cluster_key='variable_cluster'].new_cluster.spark_env_vars: skip
        -  max_concurrent_runs: replace
        -
        -
        -
        -=== Configuration changes
        -
        ->>> diff.py databricks.yml.backup databricks.yml
        ---- databricks.yml.backup
        -+++ databricks.yml
        -@@ -30,5 +30,5 @@
        -   jobs:
        -     my_job:
        --      max_concurrent_runs: 1
        -+      max_concurrent_runs: 5
        -       job_clusters:
        -         - job_cluster_key: shared
        -
        -=== Telemetry
        -
        ->>> cat [OUTPUT_DIR]/out.requests.txt
        -{
        -  "save": true,
        -  "changes_total": 7,
        -  "add_count": 3,
        -  "replace_count": 3,
        -  "remove_count": 1,
        -  "resource_changes": [
        -    {
        -      "resource_type": "jobs",
        -      "changes_count": 7,
        -      "add_count": 3,
        -      "replace_count": 3,
        -      "remove_count": 1
        -    }
        -  ],
        -  "files_changed_count": 1,
        -  "files_written_count": 1,
        -  "skipped_changes_count": 6,
        -  "state_serial": 1,
        -  "state_lineage": "[UUID]",
        -  "state_source": "local",
        -  "states_available_count": 2,
        -  "state_resource_ids": {
        -    "resource_job_ids": [
        -      "[MY_JOB_ID]"
        -    ]
        -  }
        -}
        -
         >>> [CLI] bundle destroy --auto-approve
         The following resources will be deleted:
           delete resources.jobs.my_job
        @@ -71,3 +14,5 @@
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default
         
         Destroy: 1 deleted
        +
        +Exit code: 1

latest (v1.23.0): same output as main (44f2eda)

TestAccept/bundle/config-remote-sync/variable_reference_parent/DMS=true: branch ✅ | main (44f2eda) ❌ | latest (v1.23.0) ❌

main (44f2eda):

acceptance_test.go:1342: Diff:
        --- bundle/config-remote-sync/variable_reference_parent/output.txt
        +++ /tmp/TestAcceptbundleconfig-remote-syncvariable_reference_parentD3347099602/002/output.txt
        @@ -6,64 +6,8 @@
         === Edit fields under complex variables
         
         === Detect and save all changes
        -Detected changes in 1 resource(s):
        +Error: failed to generate YAML files: failed to apply change to file [TEST_TMP_DIR]/databricks.yml for a field resources.jobs.my_job.job_clusters[1].new_cluster.custom_tags.added: failed to apply change: op add /resources/jobs/my_job/job_clusters/1/new_cluster/custom_tags/added: unexpected yaml node: scalar can not be a container
         
        -Resource: resources.jobs.my_job
        -  job_clusters[job_cluster_key='shared'].new_cluster.init_scripts[0].workspace.destination: skip
        -  job_clusters[job_cluster_key='shared'].new_cluster.spark_conf['spark.executor.memory']: skip
        -  job_clusters[job_cluster_key='variable_cluster'].new_cluster.custom_tags['added']: skip
        -  job_clusters[job_cluster_key='variable_cluster'].new_cluster.custom_tags['changed']: skip
        -  job_clusters[job_cluster_key='variable_cluster'].new_cluster.custom_tags['removed']: skip
        -  job_clusters[job_cluster_key='variable_cluster'].new_cluster.spark_env_vars: skip
        -  max_concurrent_runs: replace
        -
        -
        -
        -=== Configuration changes
        -
        ->>> diff.py databricks.yml.backup databricks.yml
        ---- databricks.yml.backup
        -+++ databricks.yml
        -@@ -30,5 +30,5 @@
        -   jobs:
        -     my_job:
        --      max_concurrent_runs: 1
        -+      max_concurrent_runs: 5
        -       job_clusters:
        -         - job_cluster_key: shared
        -
        -=== Telemetry
        -
        ->>> cat [OUTPUT_DIR]/out.requests.txt
        -{
        -  "save": true,
        -  "changes_total": 7,
        -  "add_count": 3,
        -  "replace_count": 3,
        -  "remove_count": 1,
        -  "resource_changes": [
        -    {
        -      "resource_type": "jobs",
        -      "changes_count": 7,
        -      "add_count": 3,
        -      "replace_count": 3,
        -      "remove_count": 1
        -    }
        -  ],
        -  "files_changed_count": 1,
        -  "files_written_count": 1,
        -  "skipped_changes_count": 6,
        -  "state_serial": 1,
        -  "state_lineage": "[UUID]",
        -  "state_source": "local",
        -  "states_available_count": 2,
        -  "state_resource_ids": {
        -    "resource_job_ids": [
        -      "[MY_JOB_ID]"
        -    ]
        -  }
        -}
        -
         >>> [CLI] bundle destroy --auto-approve
         The following resources will be deleted:
           delete resources.jobs.my_job
        @@ -71,3 +14,5 @@
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default
         
         Destroy: 1 deleted
        +
        +Exit code: 1
        
FAIL

latest (v1.23.0):

acceptance_test.go:1342: Diff:
        --- bundle/config-remote-sync/variable_reference_parent/output.txt
        +++ /tmp/TestAcceptbundleconfig-remote-syncvariable_reference_parentD3684698501/002/output.txt
        @@ -6,64 +6,8 @@
         === Edit fields under complex variables
         
         === Detect and save all changes
        -Detected changes in 1 resource(s):
        +Error: failed to generate YAML files: failed to apply change to file [TEST_TMP_DIR]/databricks.yml for a field resources.jobs.my_job.job_clusters[1].new_cluster.custom_tags.added: failed to apply change: op add /resources/jobs/my_job/job_clusters/1/new_cluster/custom_tags/added: unexpected yaml node: scalar can not be a container
         
        -Resource: resources.jobs.my_job
        -  job_clusters[job_cluster_key='shared'].new_cluster.init_scripts[0].workspace.destination: skip
        -  job_clusters[job_cluster_key='shared'].new_cluster.spark_conf['spark.executor.memory']: skip
        -  job_clusters[job_cluster_key='variable_cluster'].new_cluster.custom_tags['added']: skip
        -  job_clusters[job_cluster_key='variable_cluster'].new_cluster.custom_tags['changed']: skip
        -  job_clusters[job_cluster_key='variable_cluster'].new_cluster.custom_tags['removed']: skip
        -  job_clusters[job_cluster_key='variable_cluster'].new_cluster.spark_env_vars: skip
        -  max_concurrent_runs: replace
        -
        -
        -
        -=== Configuration changes
        -
        ->>> diff.py databricks.yml.backup databricks.yml
        ---- databricks.yml.backup
        -+++ databricks.yml
        -@@ -30,5 +30,5 @@
        -   jobs:
        -     my_job:
        --      max_concurrent_runs: 1
        -+      max_concurrent_runs: 5
        -       job_clusters:
        -         - job_cluster_key: shared
        -
        -=== Telemetry
        -
        ->>> cat [OUTPUT_DIR]/out.requests.txt
        -{
        -  "save": true,
        -  "changes_total": 7,
        -  "add_count": 3,
        -  "replace_count": 3,
        -  "remove_count": 1,
        -  "resource_changes": [
        -    {
        -      "resource_type": "jobs",
        -      "changes_count": 7,
        -      "add_count": 3,
        -      "replace_count": 3,
        -      "remove_count": 1
        -    }
        -  ],
        -  "files_changed_count": 1,
        -  "files_written_count": 1,
        -  "skipped_changes_count": 6,
        -  "state_serial": 1,
        -  "state_lineage": "[UUID]",
        -  "state_source": "local",
        -  "states_available_count": 2,
        -  "state_resource_ids": {
        -    "resource_job_ids": [
        -      "[MY_JOB_ID]"
        -    ]
        -  }
        -}
        -
         >>> [CLI] bundle destroy --auto-approve
         The following resources will be deleted:
           delete resources.jobs.my_job
        @@ -71,3 +14,5 @@
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default
         
         Destroy: 1 deleted
        +
        +Exit code: 1

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 4820d19

Run: 41695834586

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 271 17 7:24
✅​ aws-windows-latest-4core-16gb 273 15 6:24
✅​ azure linux-2core-8gb 270 17 9:19
✅​ azure-windows-latest-4core-16gb 272 15 6:04
✅​ gcp linux-2core-8gb 271 17 7:38
✅​ gcp-windows-latest-4core-16gb 273 15 8:39
Top 4 slowest tests (at least 2 minutes):
duration env testname
3:20 aws-windows-latest-4core-16gb TestAccept
3:17 azure-windows-latest-4core-16gb TestAccept
3:05 gcp-windows-latest-4core-16gb TestAccept
2:13 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

DABs DABs related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ilyakuz-db @eng-dev-ecosystem-bot