PESDLC-1489 StreamVerifier service #19976

savex · 2024-06-24T21:13:15Z

This implements service part of the StreamVerifier that is created in this PR.

Service will supports all commands including 'produce', 'consume' and 'atomic' as well as getting current status and active command

Backports Required

Release Notes

none

bharathv

This is marked as draft, so unsure if this is going in the current form but please see my comments.

tests/rptest/transactions/verifiers/stream_verifier.py

tests/setup.py

tests/rptest/transactions/verifiers/stream_verifier.py

savex · 2024-06-29T00:16:42Z

Most recent commit provides visibility on how very basic test structure might look like. One more thing to add is message validation option on consumption via externally pluggable validator.

vbotbuildovich · 2024-07-09T01:32:57Z

new failures in https://buildkite.com/redpanda/redpanda/builds/51238#019094e8-d681-417f-a7c3-8db584f7d721:

"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_high_watermark"
"rptest.transactions.tx_admin_api_test.TxAdminTest.test_mark_transaction_expired"

new failures in https://buildkite.com/redpanda/redpanda/builds/51587#0190bc79-b124-448a-a387-9c4f087b5635:

"rptest.transactions.stream_verifier_test.StreamVerifierTest.test_simple_produce_consume_txn_with_add_node"

new failures in https://buildkite.com/redpanda/redpanda/builds/51658#0190c184-0228-4a99-9546-e8430d8b45dd:

"rptest.transactions.stream_verifier_test.StreamVerifierTest.test_simple_produce_consume_txn_with_add_node"

new failures in https://buildkite.com/redpanda/redpanda/builds/51658#0190c184-fa72-4f55-bca5-85c2cdc49e69:

"rptest.transactions.stream_verifier_test.StreamVerifierTest.test_simple_produce_consume_txn_with_add_node"

savex · 2024-07-12T22:29:26Z

@bharathv , updated to fit simplification. Will continue on Monday to debug and polish.
Right now there is strange count mismatch happening

[INFO:2024-07-12 22:09:04,105]: RunnerClient: rptest.transactions.stream_verifier_test.StreamVerifierTest.test_streaming_upgrade: Data: None
test_id:    rptest.transactions.stream_verifier_test.StreamVerifierTest.test_simple_produce_consume_txn_with_add_node
status:     PASS
run time:   1 minute 39.690 seconds
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    rptest.transactions.stream_verifier_test.StreamVerifierTest.test_streaming_restart_unsafely_and_add
status:     IGNORE
run time:   0.000 seconds
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    rptest.transactions.stream_verifier_test.StreamVerifierTest.test_streaming_restart_safely_and_add
status:     FAIL
run time:   3 minutes 10.183 seconds

...
AssertionError: Produced/Atomic message count mismatch: 1250/1251
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    rptest.transactions.stream_verifier_test.StreamVerifierTest.test_streaming_upgrade
status:     FAIL
run time:   2 minutes 4.507 seconds

...
AssertionError: Produced/Atomic message count mismatch: 996/69

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
============================================================================================================================================================================================================================================================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.8.18
session_id:       2024-07-12--012
run time:         7 minutes 1.258 seconds
tests run:        4
passed:           1
flaky:            0
failed:           2
ignored:          1
opassed:          0
ofailed:          0
opassedfips:      0
ofailedfips:      0
============================================================================================================================================================================================================================================================================================================================

Most probably due to lack of message retry to produce after KafkaErrors on node unavailability.

vbotbuildovich · 2024-07-13T00:55:31Z

skipped ducktape retry in https://buildkite.com/redpanda/redpanda/builds/51451#0190a94f-5ebe-47b9-baed-e16504ea0f32:
pandatriage cache was not found

skipped ducktape retry in https://buildkite.com/redpanda/redpanda/builds/51451#0190a965-6c5a-43db-9d72-64e6387d099f:
pandatriage cache was not found

savex · 2024-07-15T19:44:10Z

Sample test runs using EC2

test_id:    rptest.transactions.stream_verifier_test.StreamVerifierTest.test_simple_produce_consume_txn_with_add_node
status:     PASS
run time:   2 minutes 21.611 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    rptest.transactions.stream_verifier_test.StreamVerifierTest.test_simple_produce_consume_txn_with_add_node
status:     PASS
run time:   2 minutes 41.324 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    rptest.transactions.stream_verifier_test.StreamVerifierTest.test_simple_produce_consume_txn_with_add_node
status:     PASS
run time:   2 minutes 28.480 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    rptest.transactions.stream_verifier_test.StreamVerifierTest.test_simple_produce_consume_txn_with_add_node
status:     PASS
run time:   2 minutes 16.879 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    rptest.transactions.stream_verifier_test.StreamVerifierTest.test_simple_produce_consume_txn_with_add_node
status:     PASS
run time:   2 minutes 18.292 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    rptest.transactions.stream_verifier_test.StreamVerifierTest.test_simple_produce_consume_txn_with_add_node
status:     PASS
run time:   2 minutes 24.543 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    rptest.transactions.stream_verifier_test.StreamVerifierTest.test_simple_produce_consume_txn_with_add_node
status:     PASS
run time:   2 minutes 26.633 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    rptest.transactions.stream_verifier_test.StreamVerifierTest.test_simple_produce_consume_txn_with_add_node
status:     PASS
run time:   2 minutes 18.435 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    rptest.transactions.stream_verifier_test.StreamVerifierTest.test_simple_produce_consume_txn_with_add_node
status:     PASS
run time:   2 minutes 27.722 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    rptest.transactions.stream_verifier_test.StreamVerifierTest.test_simple_produce_consume_txn_with_add_node
status:     PASS
run time:   2 minutes 17.179 seconds
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
=============================================================================================================================================================================================================================================================================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.8.18
session_id:       2024-07-15--010
run time:         24 minutes 17.711 seconds
tests run:        10
passed:           10
flaky:            0
failed:           0
ignored:          0
opassed:          0
ofailed:          0
opassedfips:      0
ofailedfips:      0
=============================================================================================================================================================================================================================================================================================================================================

vbotbuildovich · 2024-07-15T22:00:54Z

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/51539#0190b813-8c1a-4c0d-b167-1183571c629f

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/51587#0190bc7b-44ac-4632-903e-0d5b8b8b747c

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/51658#0190c184-022c-4ee8-8b50-5a9c9ccd5a2d

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/51658#0190c184-fa77-428e-8ad2-74bebbe97ffa

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/51677#0190c229-84b8-4a71-a064-26caba5d1a5c

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/51799#0190ccbe-f206-445d-802f-af11e40346d8

bharathv

bunch of nits.

bharathv · 2024-07-16T14:26:37Z

tests/rptest/transactions/verifiers/stream_verifier.py

+
+class ConsistencyViolationException(Exception):
+    pass
+


Yes, missed that. Thanks.

bharathv · 2024-07-16T14:30:27Z

tests/rptest/transactions/verifiers/stream_verifier.py

+        self.logger.debug(f"[{action}] ...{count} messages")
+        return count
+
+    def ensure_progress(self, action, delta, timeout_sec):


In these tests yes, but in case of using service for some node operations is useful. I would leave it.

bharathv · 2024-07-16T14:31:28Z

tests/rptest/transactions/verifiers/stream_verifier.py

+    def get_produce_status(self):
+        return self._get(self._get_url("produce"), raise_on_fails=False)
+
+    def get_consume_status(self):


nit on naming, make it consistent with "verify" semantics in the other class?

yes, will do

bharathv · 2024-07-16T14:32:14Z

tests/rptest/transactions/stream_verifier_test.py

+        self.target_topic = "stream_topic_dst"
+
+        # Calculated speed of producer is ~100k messages per minute, which is ~1500/sec
+        # Amtomic thread processed ~250 messages per second.


bharathv · 2024-07-16T14:33:21Z

tests/rptest/transactions/stream_verifier_test.py

+        self.verifier.remote_start_produce(self.source_topic,
+                                           self.default_message_count,
+                                           messages_per_sec=messages_per_sec)
+        self.logger.info(f"Waiting for {wait_msg_count} produces messages")


Is this missing a call to ensure progress or am I misreading?

It is in the next line: https://github.com/redpanda-data/redpanda/pull/19976/files/cf47568343cf5570ff48c14aeeae5da61a87b1db#diff-54b41e2f630fa459c2026ad573cab55e04668e9ba1bf865709c134155522fc46R79

bharathv · 2024-07-17T03:26:00Z

@savex Failure seems related.

savex · 2024-07-17T14:38:51Z

@bharathv, yes, I keep forgetting that CDT runs in docker and there should be different set of timings

savex · 2024-07-17T18:10:43Z

/cdt
release
rp_version=build
skip-units
dt-repeat=20
tests/rptest/transactions/stream_verifier_test.py

savex · 2024-07-17T18:15:48Z

/cdt
release
rp_version=build
skip-units
dt-repeat=20
tests/rptest/transactions/stream_verifier_test.py

Based on remote script that has stream verifier webservice implemented, this servie provides easy to use interface for produce, atomic consume/produce and consume actions. Basic service workflow is verifier.start verifier.update_service_config verifier.remote_start_produce verifier.remote_start_atomic verifier.remote_wait_action('produce') verifier.remote_wait_action('atomic') verifier.remote_start_consume verifier.remote_wait_action('consume') And use verifier.get_produce_status to check on offsets and/or message count

Test checks that produce/atomic/consume action can handle broker addition safely. Also, some internal unified routines implemented to make test itself look simpler. Fixed typos and remove unused exception Updated message count for non-dedicated nodes env

savex requested review from clee and bharathv June 24, 2024 21:13

bharathv reviewed Jun 25, 2024

View reviewed changes

savex force-pushed the 1489-stream-verifier-service branch 5 times, most recently from cda563c to e09ac04 Compare June 29, 2024 00:13

savex force-pushed the 1489-stream-verifier-service branch 6 times, most recently from 12fa904 to e1d15ac Compare July 8, 2024 22:54

savex marked this pull request as ready for review July 8, 2024 22:54

savex requested a review from bharathv July 8, 2024 22:54

savex force-pushed the 1489-stream-verifier-service branch from fc8c811 to d7efb91 Compare July 12, 2024 22:26

savex force-pushed the 1489-stream-verifier-service branch 2 times, most recently from c43066b to cf47568 Compare July 15, 2024 19:15

bharathv reviewed Jul 16, 2024

View reviewed changes

savex force-pushed the 1489-stream-verifier-service branch from 3bc1cc1 to a2db278 Compare July 17, 2024 18:09

savex force-pushed the 1489-stream-verifier-service branch from a2db278 to d0b7563 Compare July 17, 2024 18:14

savex requested a review from bharathv July 18, 2024 18:24

savex added 2 commits July 19, 2024 14:07

savex force-pushed the 1489-stream-verifier-service branch from d0b7563 to 5aafe8c Compare July 19, 2024 19:08

bharathv approved these changes Jul 19, 2024

View reviewed changes

savex merged commit c04ae45 into dev Jul 22, 2024
25 checks passed

savex deleted the 1489-stream-verifier-service branch July 22, 2024 14:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PESDLC-1489 StreamVerifier service #19976

PESDLC-1489 StreamVerifier service #19976

savex commented Jun 24, 2024

bharathv left a comment

savex commented Jun 29, 2024

vbotbuildovich commented Jul 9, 2024 •

edited

Loading

savex commented Jul 12, 2024

vbotbuildovich commented Jul 13, 2024 •

edited

Loading

savex commented Jul 15, 2024

vbotbuildovich commented Jul 15, 2024 •

edited

Loading

bharathv left a comment

bharathv Jul 16, 2024

savex Jul 16, 2024

bharathv Jul 16, 2024

savex Jul 16, 2024

bharathv Jul 16, 2024

savex Jul 16, 2024

bharathv Jul 16, 2024

savex Jul 16, 2024

bharathv Jul 16, 2024

savex Jul 16, 2024

bharathv commented Jul 17, 2024

savex commented Jul 17, 2024

savex commented Jul 17, 2024

savex commented Jul 17, 2024

PESDLC-1489 StreamVerifier service #19976

PESDLC-1489 StreamVerifier service #19976

Conversation

savex commented Jun 24, 2024

Backports Required

Release Notes

bharathv left a comment

Choose a reason for hiding this comment

savex commented Jun 29, 2024

vbotbuildovich commented Jul 9, 2024 • edited Loading

savex commented Jul 12, 2024

vbotbuildovich commented Jul 13, 2024 • edited Loading

savex commented Jul 15, 2024

vbotbuildovich commented Jul 15, 2024 • edited Loading

bharathv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bharathv commented Jul 17, 2024

savex commented Jul 17, 2024

savex commented Jul 17, 2024

savex commented Jul 17, 2024

vbotbuildovich commented Jul 9, 2024 •

edited

Loading

vbotbuildovich commented Jul 13, 2024 •

edited

Loading

vbotbuildovich commented Jul 15, 2024 •

edited

Loading