[FLINK-40048][e2e] Replace Kafka with filesystem connector in SqlClientITCase#28606
Merged
MartijnVisser merged 2 commits intoJul 3, 2026
Merged
Conversation
…ntITCase testMatchRecognize has been @disabled since the Kafka connector was externalized (FLINK-30859), leaving kafka-clients, the 1.17-era flink-sql-connector-kafka and a KafkaContainer behind for no active test while still requiring dependency upgrades. Read the test records from a file staged into the JobManager and TaskManager containers instead, and re-enable the test. Verify results by waiting for the bounded job to reach FINISHED rather than polling the sink file in an unbounded loop that swallowed job failures. Also drop the RegReplace function registration: its invocation moved to the Kafka-specific script in FLINK-10624 and CREATE FUNCTION is covered by the flink-end-to-end-tests-sql module, which still consumes the SqlToolbox.jar built here. Generated-by: Claude Code (Sonnet 4.6)
Collaborator
1 task
snuyanzin
reviewed
Jul 2, 2026
| "{\"timestamp\": \"2018-03-12T10:40:00Z\", \"user\": \"Bob\", \"event\": { \"type\": \"ERROR\", \"message\": \"This is an error.\"}}" | ||
| }; | ||
| sendMessages("test-json", messages); | ||
| File inputFile = new File(tempDir, "test-json.jsonl"); |
Contributor
Author
There was a problem hiding this comment.
JSON Lines, a specific format where each line contains one JSON object, separated by a newline character
snuyanzin
reviewed
Jul 2, 2026
snuyanzin
reviewed
Jul 2, 2026
github-actions
Bot
added
the
community-reviewed
Generated-by: Claude Code (Fable 5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
SqlClientITCase#testMatchRecognizehas been@Disabledsince the Kafka connector was externalized (FLINK-30859) and was never re-enabled. The module still carriedflink-sql-connector-kafka:3.0.0-1.17(a Flink 1.17-era version of the externalized connector),kafka-clients, andtestcontainers-kafka, and started a Kafka container on every test class, all for a test that never ran. This also caused recurring maintenance such as kafka-clients version bumps (see #27924).This PR re-enables the test using the built-in filesystem connector, preserving the original intent (SQL Client submission of a streaming job with event-time watermarks and MATCH_RECOGNIZE), and removes the dead Kafka infrastructure.
Brief change log
testMatchRecognizeto read the 8 test records from a JSON Lines file staged into the JobManager container (theSourceCoordinatorenumerates splits there) and each TaskManager container, using thefilesystemconnector in explicit streaming mode; remove@Disabled.verifyNumberOfResultRecords: wait for the bounded job to reachFINISHEDviaRestClusterClientand read the sink output once, instead of polling the result file in an unbounded loop that swallowed job failures. The sink flushes all records at the latest when the bounded job finishes, so a failing job now surfaces immediately with its actual state.RegReplacefunction registration: its invocation moved to the Kafka-specific script in FLINK-10624 (2018), andCREATE FUNCTIONregister-and-invoke coverage lives inflink-end-to-end-tests-sql(UsingRemoteJarITCase,CreateTableAsITCase,PlannerScalaFreeITCase), which still consumes theSqlToolbox.jarbuilt by this module.flink-sql-connector-kafka,kafka-clients, andtestcontainers-kafkafrom the pom, including thedependencyManagementpin and the sql-jars copy entry; keepcommons-codec(needed byFlinkImageBuilder).Verifying this change
This change is covered by the module's own E2E tests: all 3 tests in
SqlClientITCasepass locally under-Prun-end-to-end-tests(~75s total), including the re-enabledtestMatchRecognize, which was additionally run 3 more times in isolation to check stability.Does this pull request potentially affect one of the following parts:
@Public(Evolving): noDocumentation
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Sonnet 4.6)