[rptest] Azure CDT bringup #20749

clee · 2024-06-28T11:49:31Z

Bug fixes to tests and test infrastructure for running CDT on Azure. (Note: Currently includes commit from #18827, which this PR continues and builds upon.)

Backports Required

Release Notes

none

Improvements

replaces several instances of deprecated Cloud API usage with new Public API (because the old endpoints don't return Azure resources)
successfully installs BYOC Azure agent
allow override headers on POST requests for rpcloud_client
workaround for /snap/bin not showing up in $PATH (making az CLI not show up as an available command)
Azure support for config_profile_verify test

CLAassistant · 2024-06-28T11:49:37Z

All committers have signed the CLA.

tests/rptest/clients/kubectl.py

ivotron · 2024-06-28T19:22:52Z

what I was trying to mention during our sync today: to ensure that these changes don't break HTT tests on other cloud providers, given that these are not exercised at PR time, some manual jobs can be triggered (pointing to this branch) so that they are executed.

clee · 2024-07-02T16:53:08Z

/cdt
provider=aws

clee · 2024-07-02T17:16:48Z

/cdt
provider=aws
rp_version=build

clee · 2024-07-03T22:16:18Z

/cdt
provider=aws
rp_version=build

rpdevmp

LGTM

clee · 2024-07-08T23:46:24Z

/cdt
provider=gcp
rp_version=build

vbotbuildovich · 2024-07-08T23:46:53Z

'cdt_instance_type' and 'region' is required if 'provider' is not 'aws'

Workflow run logs.

clee · 2024-07-09T18:42:18Z

/cdt
provider=gcp
region=us-west-1
rp_version=build
cdt_instance_type=n2-highmem-4

vbotbuildovich · 2024-07-09T21:00:03Z

new failures in https://buildkite.com/redpanda/redpanda/builds/51279#0190990e-2aff-46ac-ad35-6ae316956a3a:

"rptest.tests.services_self_test.KubectlLocalOnlyTest.test_is_redpanda_pod"

new failures in https://buildkite.com/redpanda/redpanda/builds/51287#0190999a-7a83-4c90-af52-5748d8545dd1:

"rptest.tests.services_self_test.KubectlLocalOnlyTest.test_is_redpanda_pod"

new failures in https://buildkite.com/redpanda/redpanda/builds/51287#019099a1-063a-4ac5-9ffa-fdcfea89efbe:

"rptest.tests.services_self_test.KubectlLocalOnlyTest.test_is_redpanda_pod"

new failures in https://buildkite.com/redpanda/redpanda/builds/51330#01909de2-a123-4152-a8e4-074c2331b304:

"rptest.tests.e2e_iam_role_test.AWSRoleFetchTests.test_write"

new failures in https://buildkite.com/redpanda/redpanda/builds/51338#01909f2a-f14c-4d26-a834-f865e353b0a5:

"rptest.tests.e2e_iam_role_test.ShortLivedCredentialsTests.test_short_lived_credentials"

new failures in https://buildkite.com/redpanda/redpanda/builds/51557#0190b970-9cb6-4a72-b795-0e40d6b949b7:

"rptest.tests.services_self_test.KubectlLocalOnlyTest.test_is_redpanda_pod"

new failures in https://buildkite.com/redpanda/redpanda/builds/51557#0190b971-ecbc-41f8-a486-57d9306b3d22:

"rptest.tests.services_self_test.KubectlLocalOnlyTest.test_is_redpanda_pod"

new failures in https://buildkite.com/redpanda/redpanda/builds/51557#0190bcdf-f17d-4ba1-91ad-4671061be6ec:

"rptest.tests.services_self_test.KubectlLocalOnlyTest.test_is_redpanda_pod"

new failures in https://buildkite.com/redpanda/redpanda/builds/51557#0190bcdf-ebfa-469d-965a-791b1607b6ce:

"rptest.tests.services_self_test.KubectlLocalOnlyTest.test_is_redpanda_pod"

new failures in https://buildkite.com/redpanda/redpanda/builds/51683#0190c24f-3171-4b90-bd13-b157d71c7bbc:

"rptest.tests.services_self_test.KubectlLocalOnlyTest.test_is_redpanda_pod"

new failures in https://buildkite.com/redpanda/redpanda/builds/51683#0190c267-ecc5-468a-8cba-9b845265128f:

"rptest.tests.services_self_test.KubectlLocalOnlyTest.test_is_redpanda_pod"

new failures in https://buildkite.com/redpanda/redpanda/builds/52018#0190e92b-7891-47d9-a4e0-fb75f653f972:

"rptest.tests.availability_test.AvailabilityTests.test_recovery_after_catastrophic_failure"

new failures in https://buildkite.com/redpanda/redpanda/builds/52018#0190ea59-4c39-4aa7-a84f-feaa9a9dae2e:

"rptest.tests.availability_test.AvailabilityTests.test_recovery_after_catastrophic_failure"

clee · 2024-07-10T15:34:49Z

/cdt
provider=gcp
region=us-west1
rp_version=build
cdt_instance_type=n2-highmem-4

jackietung-redpanda · 2024-07-10T17:11:50Z

tests/rptest/tests/services_self_test.py

 class KubectlLocalOnlyTest(Test):

-    # @cluster(num_nodes=0)
+    @dt_cluster(num_nodes=0)


Note that without being explicit (i.e. @cluster(num_nodes=...), each test case by default asks for ALL the available nodes in session. Two additional conditions in combination causes test failure by "Test requested X nodes, but used 0":

This test should not, and does not actually use any nodes.

ducktape is sometimes invoked with --fail-bad-cluster-utilization, depending on the context. cdt_cloud.sh does NOT use --fail-bad-cluster-utilization, but other entrypoints into our DT tests often DO.

We cannot use our custom @cluster defined within our rptest framework because it requires the test to have a redpanda attribute - which we do not want (we don't want to spin up any services nor utilize any nodes).

So we fallback to using the regular @cluster decorator, to specify that we need 0 nodes for this test.

vbotbuildovich · 2024-07-11T01:21:23Z

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/51338#01909f15-275e-4af9-b361-20c1ed4c04e3

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/51557#0190b971-ecbc-41f8-a486-57d9306b3d22

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/51557#0190b971-ecbe-4968-9ead-22c220f423f9

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/51557#0190bcdf-f17d-4ba1-91ad-4671061be6ec

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/51557#0190bcdf-ebfa-469d-965a-791b1607b6ce

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/51683#0190c24f-3175-48d7-9b2b-7b699a372054

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/51683#0190c267-eccb-4fb5-ba15-2de26d525917

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/52018#0190ff3f-7af7-4bb8-9512-fc1de35fdc98

tests/rptest/clients/kubectl.py

- replaces several instances of deprecated Cloud API usage with new Public API (because the old endpoints don't return Azure resources) - successfully installs BYOC Azure agent - allow override headers on POST requests for rpcloud_client - workaround for /snap/bin not showing up in $PATH (making `az` CLI not show up as an available command) - Azure support for config_profile_verify test - Provider-specific workarounds for cases where the Azure operator has chosen to name things differently than the AWS and GCP operators

…tilization fail

savex

lgtm

nit: some minor pod name detection code flow is questionable, but it is the same on the cloud API anyway.

savex · 2024-07-18T15:11:37Z

tests/rptest/services/provider_clients/azure_client.py

+
+    def _make_client(self):
+        # TODO: Work on Azure client
+        return None


Yes, None here is a part of the flow. 'NotImplementedError' is better, though. We can improve that later

savex · 2024-07-18T15:12:22Z

tests/rptest/services/provider_clients/azure_client.py

+        return "hardcode"  # HACK
+
+    def find_vpc_peering_connection(self, state, params):
+        return None


Let us add this to the list of refactoring once this Azure thing will work.

savex · 2024-07-18T15:13:20Z

tests/rptest/services/provider_clients/rpcloud_client.py

-        resp = requests.post(f'{base_url}{endpoint}',
-                             headers=headers,
-                             **kwargs)
+        } | override_headers


This is good.

clee · 2024-07-23T19:53:18Z

/cdt
provider=gcp
region=us-west1
rp_version=build
cdt_instance_type=n2-highmem-4
tests/rptest/test_suite_cloud.yml

clee · 2024-07-24T07:35:37Z

/cdt
provider=gcp
region=us-west1
rp_repo=nightly
rp_version=latest
cdt_instance_type=n2-highmem-4
tests/rptest/test_suite_quick.yml

clee · 2024-07-24T17:14:24Z

/cdt
provider=aws
rp_repo=nightly
rp_version=latest
tests/rptest/test_suite_quick.yml

clee · 2024-07-25T05:57:49Z

AWS CDT results:

================================================================================
| SESSION REPORT (ALL TESTS)
| ducktape version: 0.8.18
| session_id:       2024-07-24--001
| run time:         679 minutes 41.394 seconds
| tests run:        1590
| passed:           1464
| flaky:            0
| failed:           17
| ignored:          2
| opassed:          98
| ofailed:          9
| opassedfips:      0
| ofailedfips:      0
| ================================================================================

clee · 2024-07-25T17:21:17Z

/cdt
provider=gcp
region=us-west1
rp_repo=nightly
rp_version=latest
cdt_instance_type=n2-highmem-4
tests/rptest/test_suite_quick.yml

clee · 2024-07-29T15:53:30Z

GCP CDT results:

================================================================================
| SESSION REPORT (ALL TESTS)
| ducktape version: 0.8.18
| session_id:       2024-07-25--001
| run time:         707 minutes 4.610 seconds
| tests run:        1590
| passed:           1462
| flaky:            0
| failed:           19
| ignored:          2
| opassed:          98
| ofailed:          9
| opassedfips:      0
| ofailedfips:      0
| ================================================================================

clee · 2024-08-02T17:08:34Z

Pulling the trigger on this merge now. If any new CDT nightly failures show up, we'll deal with them as they come.

clee requested review from savex, simonlord and rpdevmp June 28, 2024 11:49

clee force-pushed the clee/PESDLC-1432 branch from 8e0a90d to a6daf26 Compare June 28, 2024 11:50

clee requested a review from ivotron June 28, 2024 19:09

ivotron reviewed Jun 28, 2024

View reviewed changes

tests/rptest/clients/kubectl.py Show resolved Hide resolved

clee force-pushed the clee/PESDLC-1432 branch 3 times, most recently from 6abff40 to c4fadbe Compare July 2, 2024 10:47

clee force-pushed the clee/PESDLC-1432 branch from c4fadbe to 2743176 Compare July 3, 2024 18:38

rpdevmp previously approved these changes Jul 8, 2024

View reviewed changes

clee dismissed rpdevmp’s stale review via 6d7fef4 July 8, 2024 23:15

clee force-pushed the clee/PESDLC-1432 branch from ce9a286 to 6d7fef4 Compare July 8, 2024 23:15

clee marked this pull request as ready for review July 9, 2024 17:44

jackietung-redpanda reviewed Jul 10, 2024

View reviewed changes

clee requested a review from travisdowns July 11, 2024 17:21

travisdowns reviewed Jul 12, 2024

View reviewed changes

tests/rptest/clients/kubectl.py Outdated Show resolved Hide resolved

travisdowns reviewed Jul 12, 2024

View reviewed changes

tests/rptest/clients/kubectl.py Outdated Show resolved Hide resolved

travisdowns reviewed Jul 12, 2024

View reviewed changes

tests/rptest/clients/kubectl.py Show resolved Hide resolved

clee requested review from rpdevmp and jackietung-redpanda July 16, 2024 16:33

jackietung-redpanda previously approved these changes Jul 16, 2024

View reviewed changes

rpdevmp and others added 9 commits July 17, 2024 13:58

Added Azure support for cloud tests

f4b8be8

universal kubectl.is_redpanda_pod()

dc8d094

[rptest] Initial Azure machinetypes in a few more places

d5f0b28

Fix cluster id check in config_profile_verify_test

a08330e

add cluster config field to make skipping metrics health check possible

9441040

test_is_redpanda_pod - ask for 0 nodes explicitly, so we don't trip u…

d3e5b6e

…tilization fail

fix rolling restarts test (op-v1 / op-v2 trouble)

f4d0551

[rptest] Address review comments from TravisD

b9b5f85

clee force-pushed the clee/PESDLC-1432 branch from 39547af to b9b5f85 Compare July 17, 2024 18:58

savex previously approved these changes Jul 18, 2024

View reviewed changes

clee dismissed stale reviews from savex and jackietung-redpanda via 46266eb July 25, 2024 07:31

[rptest] Fix KubectlLocalOnlyTest

170faf6

clee force-pushed the clee/PESDLC-1432 branch from 46266eb to 170faf6 Compare July 25, 2024 07:35

clee requested review from jackietung-redpanda and savex July 29, 2024 19:53

jackietung-redpanda approved these changes Jul 29, 2024

View reviewed changes

clee merged commit f2cdd7d into dev Aug 2, 2024
16 of 17 checks passed

clee deleted the clee/PESDLC-1432 branch August 2, 2024 17:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[rptest] Azure CDT bringup #20749

[rptest] Azure CDT bringup #20749

clee commented Jun 28, 2024 •

edited

Loading

CLAassistant commented Jun 28, 2024 •

edited

Loading

ivotron commented Jun 28, 2024

clee commented Jul 2, 2024

clee commented Jul 2, 2024

clee commented Jul 3, 2024

rpdevmp left a comment

clee commented Jul 8, 2024

vbotbuildovich commented Jul 8, 2024

clee commented Jul 9, 2024

vbotbuildovich commented Jul 9, 2024 •

edited

Loading

clee commented Jul 10, 2024

jackietung-redpanda Jul 10, 2024

vbotbuildovich commented Jul 11, 2024 •

edited

Loading

savex left a comment

savex Jul 18, 2024

savex Jul 18, 2024

savex Jul 18, 2024

clee commented Jul 23, 2024

clee commented Jul 24, 2024

clee commented Jul 24, 2024

clee commented Jul 25, 2024 •

edited

Loading

clee commented Jul 25, 2024

clee commented Jul 29, 2024

clee commented Aug 2, 2024

[rptest] Azure CDT bringup #20749

[rptest] Azure CDT bringup #20749

Conversation

clee commented Jun 28, 2024 • edited Loading

Backports Required

Release Notes

Improvements

CLAassistant commented Jun 28, 2024 • edited Loading

ivotron commented Jun 28, 2024

clee commented Jul 2, 2024

clee commented Jul 2, 2024

clee commented Jul 3, 2024

rpdevmp left a comment

Choose a reason for hiding this comment

clee commented Jul 8, 2024

vbotbuildovich commented Jul 8, 2024

clee commented Jul 9, 2024

vbotbuildovich commented Jul 9, 2024 • edited Loading

clee commented Jul 10, 2024

jackietung-redpanda Jul 10, 2024

Choose a reason for hiding this comment

vbotbuildovich commented Jul 11, 2024 • edited Loading

savex left a comment

Choose a reason for hiding this comment

savex Jul 18, 2024

Choose a reason for hiding this comment

savex Jul 18, 2024

Choose a reason for hiding this comment

savex Jul 18, 2024

Choose a reason for hiding this comment

clee commented Jul 23, 2024

clee commented Jul 24, 2024

clee commented Jul 24, 2024

clee commented Jul 25, 2024 • edited Loading

clee commented Jul 25, 2024

clee commented Jul 29, 2024

clee commented Aug 2, 2024

clee commented Jun 28, 2024 •

edited

Loading

CLAassistant commented Jun 28, 2024 •

edited

Loading

vbotbuildovich commented Jul 9, 2024 •

edited

Loading

vbotbuildovich commented Jul 11, 2024 •

edited

Loading

clee commented Jul 25, 2024 •

edited

Loading