Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KAFKA-17609:[3/4]Convert system tests to kraft part 3 #17327

Merged

Conversation

bbejeck
Copy link
Contributor

@bbejeck bbejeck commented Sep 30, 2024

  1. Broker compatibility test
  2. Streams optimized test
  3. Streams relational smoke test
  4. Streams shutdown deadlock test

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

@github-actions github-actions bot added the tests Test fixes (including flaky tests) label Sep 30, 2024
@bbejeck
Copy link
Contributor Author

bbejeck commented Sep 30, 2024

Links to passing tests
Streams optimized test
Streams relational smoke test
Streams shutdown deadlock test

Broker compatibility test is failing but seems unrelated to the changes for switching from ZK to KRaft

@mjsax mjsax added the streams label Oct 7, 2024
@bbejeck bbejeck force-pushed the KAFKA-17609_convert_system_tests_to_kraft_part_3 branch from ada336f to 2d0494b Compare October 16, 2024 20:20
@github-actions github-actions bot added the small Small PRs label Oct 16, 2024
@@ -86,7 +86,7 @@ def __init__(self, test_context):

@cluster(num_nodes=8)
@matrix(crash=[False, True],
processing_guarantee=['exactly_once', 'exactly_once_v2'],
processing_guarantee=['exactly_once_v2'],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that we have only EOSv2 left, should we not just remove this parameter completely?

Given that exactly_once was remove from the code base a while ago already, I am wondering why we did not get any failing system test alerts? (Miss on my side to not remove this setting in one of my PRs) -- Or is this test not executed? 🤔

@@ -86,7 +86,7 @@ def __init__(self, test_context):

@cluster(num_nodes=8)
@matrix(crash=[False, True],
processing_guarantee=['exactly_once', 'exactly_once_v2'],
processing_guarantee=['exactly_once_v2'],
metadata_quorum=[quorum.isolated_kraft])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seem we can remove this parameter, too, and also switch to combined_kraft ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems you missed this one?

- Streams w/ EOS-v2 works for older brokers 2.5 (or newer)
- Streams fails fast for older brokers 0.10.0, 0.10.2, and 0.10.1
- Streams w/ EOS-v2 fails fast for older brokers 2.4 or older
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure why we would need any changes here? For compatibility testing, we should still be able to startup old Kafka with ZK and nothing should really change?

Of course, when using trunk/4.0 we would start brokers in kraft mode.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding was that all ZK was going to be completely removed maybe we can go with this as is and in a follow-up PR revisit and see what needs to be added back

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's clarify with @cmccabe before we merge this.

@bbejeck bbejeck force-pushed the KAFKA-17609_convert_system_tests_to_kraft_part_3 branch from 2d0494b to 9c024d4 Compare October 22, 2024 01:13
Copy link
Member

@mjsax mjsax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bbejeck
Copy link
Contributor Author

bbejeck commented Oct 30, 2024

Link to passing tests.

zk_to_kraft_conversion_pr_3

@bbejeck bbejeck merged commit 358d877 into apache:trunk Oct 30, 2024
3 of 4 checks passed
@bbejeck bbejeck deleted the KAFKA-17609_convert_system_tests_to_kraft_part_3 branch October 30, 2024 16:21
@bbejeck
Copy link
Contributor Author

bbejeck commented Oct 30, 2024

Merged #17327 into trunk

abhishekgiri23 pushed a commit to abhishekgiri23/kafka that referenced this pull request Nov 2, 2024
Part 3 of 4 converting streams system tests to KRaft

Reviewers: Matthias Sax <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
small Small PRs streams tests Test fixes (including flaky tests)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants