Add KillCaller feature to test server for CLI crash recovery testing #4146

varundeepsaini · 2025-12-15T15:18:29Z

Closes: #4140

Changes

Add KillCaller option to test server that terminates the CLI process when specific endpoints are hit. This enables testing crash recovery scenarios like bundle deploy interrupted mid-execution.

KillCaller specifies how many times to kill before allowing normal responses, enabling retry/recovery testing.

[[Server]]
Pattern = "GET /api/2.0/workspace/list"
KillCaller = 2  # Kill first 2 calls, then allow

Why

Enables testing CLI crash recovery scenarios (e.g., bundle deploy interrupted mid-execution).

Tests

Added acceptance/selftest/kill_caller/ tests.

varundeepsaini · 2025-12-16T14:30:04Z

@denik could you please review this pr

varundeepsaini · 2025-12-17T15:12:56Z

@denik can you please queue the integration tests

varundeepsaini · 2025-12-30T14:36:09Z

@denik bump ^^

cmd/root/root.go

acceptance/selftest/kill_caller/test.toml

acceptance/selftest/kill_caller/currentuser/script

acceptance/selftest/kill_caller/mid_request/test.toml

eng-dev-ecosystem-bot · 2025-12-30T15:31:08Z

Commit: 1739962

Run: 20754541660

	Env	❌FAIL	🟨KNOWN	💚RECOVERED	🙈SKIP	✅pass	🙈skip	Time
🟨	aws linux		15	5	2	381	670	339:48
🟨	aws windows		20		2	383	668	71:09
🟨	aws-ucws linux		11	9	2	532	547	344:46
🟨	aws-ucws windows		16	4	2	534	545	77:32
🟨	azure linux		10	5	3	381	669	355:56
🟨	azure windows		15		3	383	667	61:28
🟨	azure-ucws linux		10	5	3	528	546	306:32
❌	azure-ucws windows	2	15		3	528	544	63:34
🟨	gcp linux		10	5	3	370	675	29:02
🟨	gcp windows		15		3	372	673	25:11

24 interesting tests: 21 KNOWN, 2 FAIL, 1 SKIP

	Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
🟨	TestAccept	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K
🟨	TestAccept/bundle/deployment/bind/alert	🙈S	🙈S	🙈S	🙈S	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K
🟨	TestAccept/bundle/resources/alerts/basic	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K
🟨	TestAccept/bundle/resources/alerts/basic/DATABRICKS_BUNDLE_ENGINE=direct	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K
🟨	TestAccept/bundle/resources/alerts/basic/DATABRICKS_BUNDLE_ENGINE=terraform	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K
🟨	TestAccept/bundle/resources/alerts/with_file	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K
🟨	TestAccept/bundle/resources/alerts/with_file/DATABRICKS_BUNDLE_ENGINE=direct	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K
🟨	TestAccept/bundle/resources/alerts/with_file/DATABRICKS_BUNDLE_ENGINE=terraform	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K
❌	TestAccept/bundle/resources/dashboards/dataset-catalog-schema	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p	✅p
❌	TestAccept/bundle/resources/dashboards/dataset-catalog-schema/DATABRICKS_BUNDLE_ENGINE=direct/Ignore=databricks.yml	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p	✅p
🙈	TestAccept/bundle/resources/permissions	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions	🟨K	🟨K	🟨K	🟨K	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct	🟨K	🟨K	🟨K	🟨K
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform	🟨K	🟨K	💚R	💚R
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions	🟨K	🟨K	💚R	💚R	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct	🟨K	🟨K	💚R	💚R
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform	🟨K	🟨K	💚R	💚R
🟨	TestExport	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K
🟨	TestExportWithFileFlag	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K
🟨	TestImportDir	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K
🟨	TestImportDirDoesNotOverwrite	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K
🟨	TestImportDirWithOverwriteFlag	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K
🟨	TestImportFileFormatAuto	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K
🟨	TestImportFileFormatSource	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K

Top 25 slowest tests (at least 2 minutes):

duration	env	testname
6:34	aws-ucws windows	TestAccept/bundle/resources/synced_database_tables/basic
6:26	aws-ucws linux	TestAccept/bundle/resources/synced_database_tables/basic
5:41	aws linux	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:37	aws linux	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:34	gcp windows	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:28	aws windows	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:27	gcp windows	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:25	aws windows	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:24	gcp linux	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:16	gcp linux	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:05	azure-ucws windows	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
3:24	azure linux	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
3:13	azure-ucws linux	TestAccept/bundle/resources/synced_database_tables/basic
3:03	azure-ucws windows	TestAccept/bundle/resources/synced_database_tables/basic
2:16	aws-ucws windows	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
2:15	gcp linux	TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.9
2:11	aws windows	TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.9
2:10	aws-ucws linux	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
2:10	gcp linux	TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=yes/PY=yes/READPLAN=
2:09	aws-ucws linux	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
2:08	aws linux	TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.13
2:08	aws-ucws windows	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
2:06	gcp linux	TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=no/READPLAN=
2:03	gcp windows	TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=no/PY=no/READPLAN=
2:02	azure-ucws windows	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform

Signed-off-by: Varun Deep Saini <[email protected]>

varundeepsaini · 2025-12-30T18:27:08Z

@denik
I changed KillCaller from bool to int to support retry scenarios. Now, we can specify how many times to kill before allowing requests through (e.g., KillCaller = 3 kills the first 3 attempts, 4th succeeds).

Had a few questions,

Should I add thread safety (via atomics)?
Would a binary string be useful for granular control (e.g., "1011" = kill, allow, kill, kill)?

Signed-off-by: Varun Deep Saini <[email protected]>

denik · 2026-01-02T09:52:12Z

@denik I changed KillCaller from bool to int to support retry scenarios. Now, we can specify how many times to kill before allowing requests through (e.g., KillCaller = 3 kills the first 3 attempts, 4th succeeds).

That's a good idea.

Should I add thread safety (via atomics)?

yes. simple mutex would work.

Would a binary string be useful for granular control (e.g., "1011" = kill, allow, kill, kill)?
I think we can start with the counter and see how far we can take it.

I was thinking of more general request matcher that can match requests based on payload substring. Then we won't need predefined sequence and also be more reliable in case of unexpected retries.

varundeepsaini · 2026-01-02T10:23:56Z

@denik I can add another field for the regex string, and kill all the requests which have the body matching that regex does that work with what idea you had ? (i would still like to keep the int field as well, for easier use)

denik · 2026-01-02T10:25:17Z

@denik I can add another field for the regex string, and kill all the requests which have the body matching that regex does that work with what idea you had ? (i would still like to keep the int field as well, for easier use)

In this PR, let's keep your counter approach. Body matching is just an idea for a separate feature.

varundeepsaini · 2026-01-06T11:44:37Z

@denik bump ^^

denik

Looks great, thanks!

varundeepsaini · 2026-01-06T12:32:34Z

@denik
Now that this is done, I wanted to start with the WAL implementation
Could you do the final review of the plan ?

github-actions · 2026-01-06T16:19:53Z

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

PR number: 4146
Commit SHA: 17399629ff0ca5de8030c2e7ffce8ba87d281e48

Checks will be approved automatically on success.

denik · 2026-01-07T12:13:36Z

@varundeepsaini I noticed the new tests are flaky on my laptop. Could you check if can repro and perhaps you know what could be the reason?

% go test ./acceptance -run ^TestAccept$/^selftest$ -count=100 -failfast
--- FAIL: TestAccept (1.05s)
    acceptance_test.go:1194: [python3 /Users/denis.bilenko/work/cli-main/acceptance/install_terraform.py --targetdir /Users/denis.bilenko/work/cli-main/acceptance/build/darwin_arm64] took 60.146667ms
    acceptance_test.go:1198: [python3 /Users/denis.bilenko/work/cli-main/acceptance/install_terraform.py --targetdir /Users/denis.bilenko/work/cli-main/acceptance/build/darwin_arm64] output: Read version 1.100.0 from /Users/denis.bilenko/work/cli-main/bundle/internal/tf/codegen/schema/version.go
    acceptance_test.go:1194: [uv build --no-cache -q --wheel --out-dir /Users/denis.bilenko/work/cli-main/acceptance/build/darwin_arm64] took 413.272875ms
    acceptance_test.go:1194: [go build -o /Users/denis.bilenko/work/cli-main/acceptance/build/darwin_arm64/databricks] took 391.47125ms
    acceptance_test.go:1194: [make -s tools/yamlfmt] took 13.735083ms
    acceptance_test.go:366: Summary (dirs): 23/28/625 run/selected/total, 5 skipped
    --- FAIL: TestAccept/selftest/kill_caller/currentuser (0.00s)
        --- FAIL: TestAccept/selftest/kill_caller/currentuser/DATABRICKS_BUNDLE_ENGINE=direct (0.19s)
            prepare_server.go:269: KillCaller: killed PID 48264 (pattern: GET /api/2.0/preview/scim/v2/Me)
            acceptance_test.go:819:
                        Error Trace:    /Users/denis.bilenko/work/cli-main/libs/testdiff/testdiff.go:22
                                                                /Users/denis.bilenko/work/cli-main/acceptance/acceptance_test.go:819
                                                                /Users/denis.bilenko/work/cli-main/acceptance/acceptance_test.go:703
                                                                /Users/denis.bilenko/work/cli-main/acceptance/acceptance_test.go:359
                        Error:          Not equal:
                                        expected: "\n>>> errcode [CLI] current-user me\n[PROCESS_KILLED]\n\nExit code: [KILLED]\nScript continued after kill\n"
                                        actual  : "\n>>> errcode [CLI] current-user me\n{}\n[PROCESS_KILLED]\n\nExit code: [KILLED]\nScript continued after kill\n"

                                        Diff:
                                        --- Expected
                                        +++ Actual
                                        @@ -2,2 +2,3 @@
                                         >>> errcode [CLI] current-user me
                                        +{}
                                         [PROCESS_KILLED]
                        Test:           TestAccept/selftest/kill_caller/currentuser/DATABRICKS_BUNDLE_ENGINE=direct
                        Messages:       selftest/kill_caller/currentuser/output.txt vs /var/folders/5y/9kkdnjw91p11vsqwk0cvmk200000gp/T/TestAcceptselftestkill_callercurrentuserDATABRICKS_BUNDLE_EN3290908576/001/output.txt
FAIL
FAIL    github.com/databricks/cli/acceptance    6.282s
FAIL

varundeepsaini · 2026-01-07T12:16:03Z

Checking

varundeepsaini · 2026-01-07T13:03:35Z

@denik #4210

eng-dev-ecosystem-bot · 2026-01-07T16:05:24Z

Commit: a0aa88c

Run: 20777744743

	Env	❌FAIL	🟨KNOWN	🔄flaky	💚RECOVERED	🙈SKIP	✅pass	🙈skip	Time
🟨	aws linux		11		9	2	417	661	347:12
🟨	aws windows		16		4	2	419	659	88:45
🟨	aws-ucws linux		11		9	2	586	532	350:59
❌	aws-ucws windows	2	16		4	2	586	530	107:06
🟨	azure linux		2	7	8	3	415	660	187:18
❌	azure windows	2	15			3	417	658	82:13
🟨	azure-ucws linux		10		5	3	582	531	325:26
❌	azure-ucws windows	1	15			3	583	529	97:30
🟨	gcp linux		6		9	3	397	669	50:47
🟨	gcp windows		15			3	399	667	52:15

29 interesting tests: 17 KNOWN, 5 FAIL, 4 RECOVERED, 2 flaky, 1 SKIP

	Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
🟨	TestAccept	🟨K	🟨K	🟨K	🟨K	💚R	🟨K	🟨K	🟨K	🟨K	🟨K
🟨	TestAccept/bundle/deployment/bind/alert	🙈S	🙈S	🙈S	🙈S	🔄f	🟨K	🟨K	🟨K	🟨K	🟨K
🟨	TestAccept/bundle/resources/alerts/basic	🟨K	🟨K	🟨K	🟨K	💚R	🟨K	🟨K	🟨K	🟨K	🟨K
🟨	TestAccept/bundle/resources/alerts/basic/DATABRICKS_BUNDLE_ENGINE=direct	🟨K	🟨K	🟨K	🟨K	🔄f	🟨K	🟨K	🟨K	🟨K	🟨K
🟨	TestAccept/bundle/resources/alerts/basic/DATABRICKS_BUNDLE_ENGINE=terraform	🟨K	🟨K	🟨K	🟨K	🔄f	🟨K	🟨K	🟨K	💚R	🟨K
🟨	TestAccept/bundle/resources/alerts/with_file	🟨K	🟨K	🟨K	🟨K	💚R	🟨K	🟨K	🟨K	💚R	🟨K
🟨	TestAccept/bundle/resources/alerts/with_file/DATABRICKS_BUNDLE_ENGINE=direct	🟨K	🟨K	🟨K	🟨K	🔄f	🟨K	🟨K	🟨K	💚R	🟨K
🟨	TestAccept/bundle/resources/alerts/with_file/DATABRICKS_BUNDLE_ENGINE=terraform	🟨K	🟨K	🟨K	🟨K	🔄f	🟨K	🟨K	🟨K	💚R	🟨K
🔄	TestAccept/bundle/resources/clusters/run/spark_python_task/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p	✅p
🔄	TestAccept/bundle/resources/clusters/run/spark_python_task/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/resources/dashboards/simple_syncroot	✅p	✅p	✅p	❌F	✅p	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/resources/dashboards/simple_syncroot/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	❌F	✅p	✅p	✅p	✅p	✅p	✅p
🙈	TestAccept/bundle/resources/permissions	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
❌	TestAccept/bundle/resources/permissions/factcheck	✅p	✅p	✅p	✅p	✅p	❌F	✅p	✅p	🙈s	🙈s
❌	TestAccept/bundle/resources/permissions/factcheck/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	❌F	✅p	✅p
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions	🟨K	🟨K	🟨K	🟨K	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct	🟨K	🟨K	🟨K	🟨K
💚	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform	💚R	💚R	💚R	💚R
💚	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions	💚R	💚R	💚R	💚R	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
💚	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct	💚R	💚R	💚R	💚R
💚	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform	💚R	💚R	💚R	💚R
❌	TestAccept/ssh/connection	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p	✅p
🟨	TestExport	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K
🟨	TestExportWithFileFlag	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K
🟨	TestImportDir	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K
🟨	TestImportDirDoesNotOverwrite	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K
🟨	TestImportDirWithOverwriteFlag	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K
🟨	TestImportFileFormatAuto	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K
🟨	TestImportFileFormatSource	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K	💚R	🟨K

Top 50 slowest tests (at least 2 minutes):

duration	env	testname
16:38	azure linux	TestAccept/bundle/resources/permissions/factcheck/DATABRICKS_BUNDLE_ENGINE=terraform
13:42	gcp windows	TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
13:24	azure linux	TestAccept/bundle/resources/alerts/basic/DATABRICKS_BUNDLE_ENGINE=terraform
12:00	gcp windows	TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
11:46	azure-ucws linux	TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
11:44	gcp windows	TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
11:41	aws-ucws linux	TestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform
11:31	aws-ucws windows	TestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform
11:30	azure-ucws windows	TestAccept/bundle/resources/permissions/factcheck/DATABRICKS_BUNDLE_ENGINE=terraform
11:14	aws windows	TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
11:04	aws windows	TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
10:58	aws linux	TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
10:50	aws-ucws linux	TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
10:44	aws-ucws linux	TestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=direct
10:30	azure-ucws linux	TestSparkJarTaskDeployAndRunOnVolumes/Databricks_Runtime_13.3_LTS
10:28	gcp windows	TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
10:27	aws-ucws windows	TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
10:20	aws-ucws windows	TestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=direct
10:17	gcp linux	TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
10:16	gcp linux	TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
10:08	aws windows	TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
9:00	aws windows	TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=SINGLE_USER
8:53	azure-ucws windows	TestAccept/bundle/integration_whl/custom_params/DATABRICKS_BUNDLE_ENGINE=terraform
8:52	aws windows	TestAccept/bundle/integration_whl/custom_params/DATABRICKS_BUNDLE_ENGINE=terraform
8:30	aws windows	TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
8:29	aws-ucws windows	TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=direct
8:27	azure linux	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
8:27	azure-ucws windows	TestAccept/bundle/integration_whl/custom_params/DATABRICKS_BUNDLE_ENGINE=direct
8:15	aws-ucws windows	TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=USER_ISOLATION
8:14	gcp windows	TestAccept/bundle/integration_whl/custom_params/DATABRICKS_BUNDLE_ENGINE=terraform
8:13	aws linux	TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=USER_ISOLATION
8:13	aws-ucws windows	TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=SINGLE_USER
8:10	aws-ucws linux	TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=USER_ISOLATION
8:06	gcp windows	TestAccept/bundle/integration_whl/custom_params/DATABRICKS_BUNDLE_ENGINE=direct
8:05	azure-ucws windows	TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
8:05	gcp linux	TestAccept/bundle/run/app-with-job/DATABRICKS_BUNDLE_ENGINE=terraform
8:03	gcp windows	TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=USER_ISOLATION
8:03	gcp linux	TestSparkJarTaskDeployAndRunOnWorkspace/Databricks_Runtime_15.4_LTS
8:01	azure windows	TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
8:00	aws-ucws linux	TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=USER_ISOLATION
7:55	aws-ucws linux	TestSparkJarTaskDeployAndRunOnVolumes/Databricks_Runtime_15.4_LTS
7:52	aws-ucws windows	TestSparkJarTaskDeployAndRunOnVolumes/Databricks_Runtime_15.4_LTS
7:50	aws linux	TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
7:46	gcp linux	TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=USER_ISOLATION
7:46	azure linux	TestSparkJarTaskDeployAndRunOnWorkspace/Databricks_Runtime_15.4_LTS
7:45	aws-ucws windows	TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
7:45	azure-ucws linux	TestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform
7:44	aws-ucws linux	TestSparkJarTaskDeployAndRunOnVolumes/Databricks_Runtime_13.3_LTS
7:44	aws windows	TestSparkJarTaskDeployAndRunOnWorkspace/Databricks_Runtime_14.3_LTS
7:43	aws windows	TestSparkJarTaskDeployAndRunOnWorkspace/Databricks_Runtime_15.4_LTS

varundeepsaini requested review from andrewnester, anton-107, denik, jefferycheng1, kanterov, lennartkats-db, pietern and shreyas-goenka as code owners December 15, 2025 15:18

Add KillCaller feature to test server for CLI crash recovery testing

3a3b6f5

varundeepsaini force-pushed the ft/server-crash-acceptance branch from 89e1011 to 3a3b6f5 Compare December 15, 2025 15:20

Removed other signals and now only killing is supported

617d1d3

varundeepsaini force-pushed the ft/server-crash-acceptance branch 2 times, most recently from debc6c8 to bcc288c Compare December 17, 2025 08:02

Fixed windows tests

ba053c3

varundeepsaini force-pushed the ft/server-crash-acceptance branch from bcc288c to ba053c3 Compare December 17, 2025 08:04

denik reviewed Dec 30, 2025

View reviewed changes

cmd/root/root.go Outdated Show resolved Hide resolved

acceptance/selftest/kill_caller/test.toml Show resolved Hide resolved

acceptance/selftest/kill_caller/currentuser/script Show resolved Hide resolved

acceptance/selftest/kill_caller/mid_request/test.toml Outdated Show resolved Hide resolved

Updated KillCaller to int and addressed comments

8bd1547

Signed-off-by: Varun Deep Saini <[email protected]>

cleanup

1ad08e2

Signed-off-by: Varun Deep Saini <[email protected]>

Added mutexes and a test

e0d4150

varundeepsaini requested a review from denik January 2, 2026 11:29

denik approved these changes Jan 6, 2026

View reviewed changes

denik enabled auto-merge January 6, 2026 12:25

Merge branch 'main' into ft/server-crash-acceptance

1739962

denik disabled auto-merge January 7, 2026 10:03

denik merged commit a0aa88c into databricks:main Jan 7, 2026
20 of 21 checks passed

varundeepsaini deleted the ft/server-crash-acceptance branch January 7, 2026 10:09

Add KillCaller feature to test server for CLI crash recovery testing #4146

Add KillCaller feature to test server for CLI crash recovery testing #4146

Uh oh!

Conversation

varundeepsaini commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why

Tests

Uh oh!

varundeepsaini commented Dec 16, 2025

Uh oh!

varundeepsaini commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

varundeepsaini commented Dec 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented Dec 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

varundeepsaini commented Dec 30, 2025

Uh oh!

denik commented Jan 2, 2026

Uh oh!

varundeepsaini commented Jan 2, 2026

Uh oh!

denik commented Jan 2, 2026

Uh oh!

varundeepsaini commented Jan 6, 2026

Uh oh!

denik left a comment

Choose a reason for hiding this comment

Uh oh!

varundeepsaini commented Jan 6, 2026

Uh oh!

github-actions bot commented Jan 6, 2026

Uh oh!

Uh oh!

denik commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

varundeepsaini commented Jan 7, 2026

Uh oh!

varundeepsaini commented Jan 7, 2026

Uh oh!

eng-dev-ecosystem-bot commented Jan 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

varundeepsaini commented Dec 15, 2025 •

edited

Loading

varundeepsaini commented Dec 17, 2025 •

edited

Loading

eng-dev-ecosystem-bot commented Dec 30, 2025 •

edited

Loading

denik commented Jan 7, 2026 •

edited

Loading